From fa18cabbecf958e3b093ef4113cc2a5ab4331e79 Mon Sep 17 00:00:00 2001 From: saileshwar-skyflow <156889717+saileshwar-skyflow@users.noreply.github.com> Date: Fri, 23 Oct 2026 15:34:38 +0530 Subject: [PATCH 01/11] SK-3225: Split SDK into common / skyvault / flowvault; add FlowVault Node SDK (#348) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * SK-3225: Split SDK into common/skyvault/flowvault; add FlowVault Node SDK Restructure the single `skyflow-node` package into an npm-workspaces monorepo mirroring the Java SDK split (common / skyvault / flowvault). - common (private, bundled into each published package): the contract-agnostic core — errors, utils, jwt, logging, service-account auth, config/credentials, generic validations, mutable SDK identity, Base* request/response/options classes, the BaseSkyflow facade base, and the generated auth client. Builds and tests in isolation. - skyvault -> skyflow-node (2.x): the existing contract, public API unchanged (74 exports, byte-for-byte). Its Skyflow/models extend the common Base* classes; keeps connections, detect and its data-plane generated client. - flowvault -> skyflow-flowvault-node (1.x): new unary v2 contract (insert/get/update/delete/query/detokenize/getTokens/uploadFiles/deleteFiles) wired to the generated v2 client; reconciled with the Java/Python flowvault SDKs. Each layer re-exports common and bundles it (tsup) into a self-contained tarball (verified via npm pack); the mutable SDK identity makes each product self-report its own name@version. Independent versioning: plain-semver tags publish skyflow-node, flowvault/* tags publish skyflow-flowvault-node. Tests: common 232, skyvault 878 (backward compatibility preserved), flowvault 132 (~99.85% coverage). Per-package api-extractor contract snapshots gate the public API of both SDKs. PR CI builds and tests both SDKs with one command. Docs: docs/sdk-split-plan.md. Co-Authored-By: Claude Opus 4.8 * SK-3225: fix PR CI (eslint ignores + gitleaks allowlist for generated/dist) - eslint.config.mjs: ignore all generated dirs (**/_generated_/**, legacy leading-space ** / _generated_/**), all workspace test dirs (**/test/**), and build output across workspaces — the old ignores only matched the single-package paths. - Replace three banned magic-string comparisons in hand-written code (typeof === "symbol"/"function") with equivalent non-literal checks. - Rule/gitleaks.toml: allowlist the maintainer-owned generated dirs (example JWTs) and the bundled dist/ output. Co-Authored-By: Claude Opus 4.8 * SK-3225: fix skyvault jest — don't babel-transform compiled common/lib CI resolves the skyflow-common symlink to its realpath (outside node_modules), so babel-jest was transforming the compiled common/lib and injecting core-js (an undeclared dep) -> suites failed to run. Add `common/lib` to transformIgnorePatterns so the already-CJS common package is loaded as-is. Co-Authored-By: Claude Opus 4.8 * SK-3225: add flowvault branch-coverage tests to clear codecov/project Covers the defensive edge branches the codecov/project gate flagged (net-new flowvault code dipped overall coverage 0.44% vs base): - empty/absent `records`/`response` bodies map to [] across all ops - per-record errors with no x-request-id header -> requestId null - getRequestId with an absent raw response - uploadFiles: non-Error PUT rejection, no-name/trailing-dot content-type, response record with no matching prepared columns - VaultClient: default log level, unrecognized cached auth type flowvault branches 92.51% -> 96.89% (statements/lines 100%); 132 -> 153 tests. No source/generated/config changes. Co-Authored-By: Claude Opus 4.8 * SK-3225: flowvault insert/update — per-record table + separate tokens/data Aligns the Node flowvault insert/update contract with the Java/Python SDKs: - InsertRecord/UpdateRecord gain an optional per-record `table`; when any record sets it, a single insert/update call targets multiple tables (the generated InsertRecordData/UpdateRecordData already carry tableName). - Insert/Update response records now expose `tokens` and `data` separately instead of a single merged `fields`. Updated unit tests (incl. multi-table coverage) and the api-report snapshot. flowvault 156 tests pass (100% lines, branches 96.97%); common 232, skyvault 878 unchanged. Co-Authored-By: Claude Opus 4.8 * SK-3225: add flowvault README + samples (parity with python flowvault) Expand flowvault/README.md into a full SDK guide (overview, install, auth, client init, all 9 operations, custom headers, error handling) and add flowvault/samples/ — one runnable TypeScript example per operation under vault-api/, plus service-account token-generation samples — mirroring the skyflow-flowvault-python samples. All samples type-check in strict mode against the built SDK types. flowvault-only; skyvault and common untouched. Co-Authored-By: Claude Opus 4.8 * SK-3225: flowvault timeout/retry config + wire custom request headers Config (parity with python flowvault, limited to what the Node generated client can honor): - VaultConfig gains optional timeout (seconds), maxRetries, and vaultUrl; the flowvault VaultClient forwards timeout/maxRetries as the generated request's timeoutInSeconds/maxRetries, and vaultUrl overrides the derived URL. (connect/read/write timeouts and retry-delay knobs are fixed in generated code.) Custom headers: - options.setRequestHeaders(...) is now actually wired — merged into every operation's request headers. Added the missing options params to delete, query, getTokens, uploadFiles, deleteFiles so headers work on all 9 ops. Samples/README updated; added tests (162 pass, 100% stmts/lines). flowvault + the shared VaultConfig only; skyvault untouched; api-report unchanged. Co-Authored-By: Claude Opus 4.8 * SK-3225: flowvault parity — per-record insert upsert + multi-table get fields Closes two parity gaps vs the Java/Python flowvault SDKs (generated client already supports both; hand-written layer only): - InsertRecord gains optional per-record `upsert`; when any record sets it, each record sends its own upsert (falling back to the request-level one) and the request-level upsert is dropped. - GetRecord (multi-table get) gains optional `columns` / `columnRedactions` / `uniqueValues`, wired per record. Added tests (164 pass, 100% lines); api-report regenerated. flowvault-only; skyvault and common untouched. Co-Authored-By: Claude Opus 4.8 * SK-3225: flowvault update updateType + align README/samples with python - UpdateRequest gains an optional request-level `updateType` ("UPDATE" | "REPLACE"), wired to the generated UpdateRequest.updateType (no generated change needed). Closes the last parity gap vs Java/Python flowvault. - README + samples updated to match the python flowvault SDK: per-record `upsert` on insert, multi-table get per-record columns/redactions/uniqueValues, and update `updateType`. 166 tests pass (100% lines); api-report regenerated; samples type-check. flowvault-only; skyvault/common untouched. Co-Authored-By: Claude Opus 4.8 * SK-3225: fix flowvault vault URL — use skyvault.* host, not vault.* FlowVault (v3) is served from {clusterId}.skyvault.skyflowapis.{dev|com| -preview.com|tech}, a different subdomain than the shared getVaultURL (vault.*, used by skyvault/v2). The facade was reusing getVaultURL, so every call 404'd. Added a flowvault-local getFlowVaultURL with the four confirmed skyvault.* hosts and used it in buildVaultEntry. flowvault-only; skyvault/common untouched. Co-Authored-By: Claude Opus 4.8 * SK-3225: fix flowvault insert/update table placement + per-record error extraction Two bugs surfaced running the e2e against the live vault: - Insert/update sent tableName at BOTH the request level and per-record when any record carried its own table, which the API rejects (400 "table name ... at one place"). Now the request-level tableName is omitted whenever per-record tables are used (table lives in exactly one place), matching Java/Python. - A per-record failure returned with a 4xx HTTP status (e.g. delete/get/update of a non-existent id -> 404 whose body carries a records/response array) was thrown as a whole-call error. Now those per-record records are surfaced as a normal response (via unaryError), and genuine whole-call errors still throw. Added error-extraction tests for insert/get/update/delete/detokenize/getTokens/ deleteFiles. 174 tests pass (99.4% lines). flowvault-only; skyvault/common untouched. Co-Authored-By: Claude Opus 4.8 * SK-3225: flowvault get — allow empty ids when uniqueValues are set validateGetRequest rejected empty ids even for a get-by-unique-value, where the lookup is by GetOptions.setUniqueValues (not ids). It now accepts a single-table get with empty ids as long as uniqueValues are provided on the options ("ids OR uniqueValues", matching Java/Python flowvault). 175 tests pass. flowvault-only; skyvault/common untouched; api-report unchanged. Co-Authored-By: Claude Opus 4.8 * SK-3225: flowvault custom headers — typed interceptor + CustomHeaderKey Replace the arbitrary setRequestHeaders(Record) on options with the same typed mechanism the Python and Java FlowVault SDKs expose: a RequestInterceptor set via options.setInterceptor(), which receives a RequestContext and may attach only the supported headers in CustomHeaderKey (x-skyflow-account-id, x-skyflow-account-name, x-request-id). The context carries the operation name (INSERT, GET, ...) so headers can vary per call. - New: CustomHeaderKey enum, RequestContext (+ RequestInterceptor type), FlowVaultOptions base carrying the interceptor. - All 9 *Options now extend FlowVaultOptions; controller.requestOptions runs the interceptor per operation and maps CustomHeaderKey -> wire header. - README, sample, and tests updated; contract snapshot regenerated. common and skyvault are untouched. Co-Authored-By: Claude Opus 4.8 * SK-3225: flowvault docs — vaultUrl override uses the skyvault.* host The vaultUrl config overrides the URL the SDK actually calls, and FlowVault is served from the skyvault.* subdomain (getFlowVaultURL), not vault.*. Fix the two override examples so a user who copies them points at the right host. The clusterId derivation comments keep vault.skyflowapis.com to match the Python SDK (that is where the clusterId is read from). --- .github/workflows/beta-release.yml | 4 +- .github/workflows/common-ci.yml | 7 +- .github/workflows/common-release.yml | 90 +- .github/workflows/contract-tests.yml | 18 +- .github/workflows/flowvault-beta-release.yml | 18 + .github/workflows/flowvault-release.yml | 19 + .github/workflows/internal-release.yml | 9 +- .github/workflows/release.yml | 4 +- .gitignore | 5 +- Rule/gitleaks.toml | 7 + api-extractor.json | 47 - common/.babelrc | 14 + common/babel.config.js | 32 + common/jest.config.js | 12 + common/package.json | 23 + common/src/_generated_/rest/Client.ts | 119 ++ .../src/_generated_/rest/api}/client/index.ts | 0 .../client/requests/PatchV2VaultsIdRequest.ts | 9 + .../rest/api/client/requests/index.ts | 1 + .../rest/api/errors/BadRequestError.ts | 18 + .../rest/api/errors/NotFoundError.ts | 18 + .../rest/api/errors/UnauthorizedError.ts | 18 + .../src/_generated_/rest/api/errors/index.ts | 3 + common/src/_generated_/rest/api/index.ts | 4 + .../resources/authentication/client/Client.ts | 137 ++ .../resources/authentication/client/index.ts | 0 .../client/requests/V1GetAuthTokenRequest.ts | 25 + .../authentication/client/requests/index.ts | 0 .../api/resources/authentication/index.ts | 0 .../_generated_/rest/api/resources/index.ts | 2 + .../rest/api/types/GooglerpcStatus.ts | 0 .../rest/api/types/ProtobufAny.ts | 0 .../rest/api/types/V1GetAuthTokenResponse.ts | 0 .../src/_generated_/rest/api/types/index.ts | 3 + .../_generated_}/rest/core/auth/BasicAuth.ts | 0 .../rest/core/auth/BearerToken.ts | 0 .../src/_generated_}/rest/core/auth/index.ts | 0 .../rest/core/fetcher/APIResponse.ts | 0 .../_generated_}/rest/core/fetcher/Fetcher.ts | 0 .../rest/core/fetcher/HttpResponsePromise.ts | 0 .../rest/core/fetcher/RawResponse.ts | 0 .../rest/core/fetcher/Supplier.ts | 0 .../rest/core/fetcher/createRequestUrl.ts | 0 .../rest/core/fetcher/getFetchFn.ts | 0 .../rest/core/fetcher/getHeader.ts | 0 .../rest/core/fetcher/getRequestBody.ts | 0 .../rest/core/fetcher/getResponseBody.ts | 0 .../_generated_}/rest/core/fetcher/index.ts | 0 .../rest/core/fetcher/makeRequest.ts | 0 .../rest/core/fetcher/requestWithRetries.ts | 0 .../_generated_}/rest/core/fetcher/signals.ts | 0 .../Node18UniversalStreamWrapper.ts | 259 ++++ .../stream-wrappers/NodePre18StreamWrapper.ts | 0 .../stream-wrappers/UndiciStreamWrapper.ts | 244 +++ .../stream-wrappers/chooseStreamWrapper.ts | 0 common/src/_generated_/rest/core/index.ts | 3 + .../src/_generated_}/rest/core/json.ts | 0 .../_generated_}/rest/core/runtime/index.ts | 0 .../_generated_}/rest/core/runtime/runtime.ts | 0 common/src/_generated_/rest/environments.ts | 10 + .../_generated_}/rest/errors/SkyflowError.ts | 0 .../rest/errors/SkyflowTimeoutError.ts | 0 .../src/_generated_}/rest/errors/index.ts | 0 .../src/_generated_}/rest/index.ts | 0 .../src/_generated_}/rest/version.ts | 0 common/src/base/options.ts | 11 + common/src/base/request.ts | 14 + common/src/base/response.ts | 14 + .../src}/config/connection/index.ts | 0 .../src}/config/credentials/index.ts | 0 common/src/config/vault/index.ts | 18 + {src => common/src}/error/codes/index.ts | 22 +- {src => common/src}/error/index.ts | 0 common/src/error/messages/index.ts | 252 +++ common/src/identity.ts | 15 + common/src/index.ts | 82 + .../src}/service-account/client/index.ts | 2 +- {src => common/src}/service-account/index.ts | 6 +- common/src/skyflow/base.ts | 219 +++ common/src/types/index.ts | 66 + {src => common/src}/utils/index.ts | 12 +- {src => common/src}/utils/jwt-utils/index.ts | 0 {src => common/src}/utils/logs/index.ts | 0 common/src/validations/index.ts | 380 +++++ .../test}/demo-credentials/empty.json | 0 .../test}/demo-credentials/invalidJson.json | 0 .../test}/demo-credentials/noClientId.json | 0 .../test}/demo-credentials/noKeyId.json | 0 .../test}/demo-credentials/noPrivateKey.json | 0 .../test}/demo-credentials/noTokenURI.json | 0 .../test}/demo-credentials/valid.json | 0 .../test}/error/skyflow-error.test.js | 0 .../test}/service-account/client.test.js | 4 +- .../test}/service-account/token.test.js | 0 .../test/utils}/jwt.test.js | 2 +- .../vault => common/test}/utils/utils.test.js | 44 +- common/tsconfig.json | 34 + docs/sdk-split-plan.md | 186 +++ eslint.config.mjs | 18 +- flowvault/.gitignore | 3 + flowvault/LICENSE | 21 + flowvault/README.md | 348 +++++ flowvault/_generated_/rest/src/Client.ts | 142 ++ .../_generated_/rest/src/api}/client/index.ts | 0 .../client/requests/PatchV2VaultsIdRequest.ts | 9 + .../rest/src/api/client/requests/index.ts | 1 + .../rest/src/api/errors/BadRequestError.ts | 19 + .../rest/src/api/errors/ForbiddenError.ts | 19 + .../src}/api/errors/InternalServerError.ts | 0 .../rest/src/api/errors/NotFoundError.ts | 19 + .../src/api/errors/TooManyRequestsError.ts | 19 + .../rest/src/api/errors/UnauthorizedError.ts | 19 + .../_generated_/rest/src/api/errors/index.ts | 6 + flowvault/_generated_/rest/src/api/index.ts | 4 + .../src/api/resources/files/client/Client.ts | 290 ++++ .../src}/api/resources/files/client/index.ts | 0 .../client/requests/FileDeleteRequest.ts | 23 + .../client/requests/FileUploadRequest.ts | 28 + .../resources/files/client/requests/index.ts | 2 + .../rest/src/api/resources/files}/index.ts | 0 .../rest/src/api/resources/index.ts | 9 + .../src/api/resources/query/client/Client.ts | 154 ++ .../src/api/resources/query}/client/index.ts | 0 .../client/requests/ExecuteQueryRequest.ts | 17 + .../resources/query/client/requests/index.ts | 1 + .../rest/src/api/resources/query}/index.ts | 0 .../api/resources/records/client/Client.ts | 541 +++++++ .../api/resources/records}/client/index.ts | 0 .../records/client/requests/DeleteRequest.ts | 22 + .../records/client/requests/GetRequest.ts | 32 + .../records/client/requests/InsertRequest.ts | 83 + .../records/client/requests/UpdateRequest.ts | 48 + .../records/client/requests/index.ts | 4 + .../rest/src/api/resources/records}/index.ts | 0 .../records/types/UpdateRequestUpdateType.ts | 12 + .../src/api/resources/records/types/index.ts | 1 + .../src/api/resources/tokens/client/Client.ts | 265 ++++ .../src/api/resources/tokens}/client/index.ts | 0 .../client/requests/DetokenizeRequest.ts | 21 + .../requests/GetTokensFromValuesRequest.ts | 19 + .../resources/tokens/client/requests/index.ts | 2 + .../rest/src/api/resources/tokens}/index.ts | 0 .../rest/src/api/types/ColumnRedactions.ts | 10 + .../rest/src/api/types/DeleteResponse.ts | 10 + .../src/api/types/DeleteResponseObject.ts | 12 + .../rest/src/api/types/DetokenizeResponse.ts | 10 + .../src/api/types/DetokenizeResponseObject.ts | 19 + .../rest/src}/api/types/ErrorResponse.ts | 0 .../rest/src/api/types/ErrorResponseError.ts | 14 + .../api/types/ExecuteQueryRecordResponse.ts | 8 + .../src/api/types/ExecuteQueryResponse.ts | 11 + .../api/types/ExecuteQueryResponseMetadata.ts | 11 + .../rest/src/api/types/FileDeleteRecord.ts | 16 + .../rest/src/api/types/FileDeleteResponse.ts | 10 + .../src/api/types/FileDeleteResponseObject.ts | 16 + .../rest/src/api/types/FileUploadColumn.ts | 10 + .../rest/src/api/types/FileUploadRecord.ts | 14 + .../rest/src/api/types/FileUploadResponse.ts | 10 + .../src/api/types/FileUploadResponseObject.ts | 16 + .../rest/src/api/types/GetRequestData.ts | 18 + .../rest/src/api/types/GetResponse.ts | 10 + .../types/GetTokensFromValuesRequestObject.ts | 11 + .../api/types/GetTokensFromValuesResponse.ts | 10 + .../rest/src/api/types/GoogleProtobufValue.ts | 8 + .../rest/src/api/types/HttpCode.ts | 8 + .../rest/src/api/types/InsertRecordData.ts | 15 + .../rest/src/api/types/InsertResponse.ts | 10 + .../src/api/types/RecordResponseObject.ts | 20 + .../src/api/types/TokenGroupRedactions.ts | 10 + .../src/api/types/TokenizeResponseObject.ts | 17 + .../rest/src/api/types/UniqueValue.ts | 8 + .../rest/src/api/types/UpdateRecordData.ts | 18 + .../api/types/UpdateRecordDataUpdateType.ts | 12 + .../rest/src/api/types/UpdateResponse.ts | 10 + .../_generated_/rest/src/api/types/Upsert.ts | 15 + .../rest/src/api/types/UpsertUpdateType.ts | 12 + .../_generated_/rest/src/api/types/index.ts | 34 + .../rest/src/core/auth/BasicAuth.ts | 31 + .../rest/src/core/auth/BearerToken.ts | 15 + .../_generated_/rest/src/core/auth/index.ts | 2 + .../rest/src/core/fetcher/APIResponse.ts | 23 + .../rest/src/core/fetcher/Fetcher.ts | 151 ++ .../src/core/fetcher/HttpResponsePromise.ts | 116 ++ .../rest/src/core/fetcher/RawResponse.ts | 59 + .../rest/src/core/fetcher/Supplier.ts | 11 + .../rest/src/core/fetcher/createRequestUrl.ts | 10 + .../rest/src/core/fetcher/getFetchFn.ts | 25 + .../rest/src/core/fetcher/getHeader.ts | 8 + .../rest/src/core/fetcher/getRequestBody.ts | 16 + .../rest/src/core/fetcher/getResponseBody.ts | 34 + .../rest/src/core/fetcher/index.ts | 8 + .../rest/src/core/fetcher/makeRequest.ts | 44 + .../src/core/fetcher/requestWithRetries.ts | 33 + .../rest/src/core/fetcher/signals.ts | 38 + .../Node18UniversalStreamWrapper.ts | 259 ++++ .../stream-wrappers/NodePre18StreamWrapper.ts | 107 ++ .../stream-wrappers/UndiciStreamWrapper.ts | 244 +++ .../stream-wrappers/chooseStreamWrapper.ts | 34 + flowvault/_generated_/rest/src/core/index.ts | 3 + flowvault/_generated_/rest/src/core/json.ts | 27 + .../rest/src/core/runtime/index.ts | 1 + .../rest/src/core/runtime/runtime.ts | 131 ++ .../_generated_/rest/src/environments.ts | 10 + .../rest/src/errors/SkyflowError.ts | 55 + .../rest/src/errors/SkyflowTimeoutError.ts | 10 + .../_generated_/rest/src/errors/index.ts | 2 + flowvault/_generated_/rest/src/index.ts | 4 + flowvault/_generated_/rest/src/version.ts | 1 + flowvault/api-extractor.json | 32 + .../api-report/skyflow-flowvault-node.api.md | 618 ++++++++ flowvault/jest.config.js | 19 + flowvault/package.json | 61 + flowvault/samples/README.md | 66 + .../scoped-token-generation-example.ts | 46 + .../signed-token-generation-example.ts | 52 + .../token-generation-example.ts | 38 + .../vault-api/custom-header-example.ts | 69 + flowvault/samples/vault-api/delete-files.ts | 60 + flowvault/samples/vault-api/delete-records.ts | 57 + .../samples/vault-api/detokenize-records.ts | 56 + flowvault/samples/vault-api/get-records.ts | 66 + flowvault/samples/vault-api/get-tokens.ts | 55 + flowvault/samples/vault-api/insert-records.ts | 74 + flowvault/samples/vault-api/query-records.ts | 56 + .../timeout-and-retry-config-example.ts | 62 + flowvault/samples/vault-api/update-record.ts | 62 + flowvault/samples/vault-api/upload-files.ts | 64 + flowvault/src/client/index.ts | 224 +++ flowvault/src/controller/vault/index.ts | 863 +++++++++++ flowvault/src/enums/custom-header-key.ts | 14 + flowvault/src/index.ts | 73 + flowvault/src/model/options/base.ts | 24 + .../src/model/options/delete-files/index.ts | 10 + flowvault/src/model/options/delete/index.ts | 10 + .../src/model/options/detokenize/index.ts | 10 + .../src/model/options/get-tokens/index.ts | 10 + flowvault/src/model/options/get/index.ts | 57 + flowvault/src/model/options/insert/index.ts | 21 + flowvault/src/model/options/query/index.ts | 10 + flowvault/src/model/options/update/index.ts | 10 + .../src/model/options/upload-files/index.ts | 10 + flowvault/src/model/request-context/index.ts | 55 + .../src/model/request/delete-files/index.ts | 22 + flowvault/src/model/request/delete/index.ts | 43 + .../src/model/request/detokenize/index.ts | 32 + .../src/model/request/get-tokens/index.ts | 22 + flowvault/src/model/request/get/index.ts | 52 + flowvault/src/model/request/insert/index.ts | 32 + flowvault/src/model/request/query/index.ts | 21 + flowvault/src/model/request/update/index.ts | 43 + .../src/model/request/upload-files/index.ts | 22 + .../src/model/response/delete-files/index.ts | 14 + flowvault/src/model/response/delete/index.ts | 14 + .../src/model/response/detokenize/index.ts | 14 + .../src/model/response/get-tokens/index.ts | 14 + flowvault/src/model/response/get/index.ts | 14 + flowvault/src/model/response/insert/index.ts | 14 + flowvault/src/model/response/query/index.ts | 30 + flowvault/src/model/response/update/index.ts | 14 + .../src/model/response/upload-files/index.ts | 14 + flowvault/src/skyflow/index.ts | 73 + flowvault/src/types/index.ts | 187 +++ flowvault/src/validations/index.ts | 213 +++ flowvault/src/version.ts | 4 + flowvault/test/client/client.test.ts | 205 +++ .../test/controller/delete-files.test.ts | 122 ++ flowvault/test/controller/delete.test.ts | 78 + flowvault/test/controller/detokenize.test.ts | 108 ++ .../test/controller/error-records.test.ts | 122 ++ flowvault/test/controller/get-tokens.test.ts | 83 + flowvault/test/controller/get.test.ts | 153 ++ flowvault/test/controller/insert.test.ts | 171 ++ flowvault/test/controller/query.test.ts | 59 + .../test/controller/request-options.test.ts | 114 ++ flowvault/test/controller/update.test.ts | 121 ++ .../test/controller/upload-files.test.ts | 316 ++++ flowvault/test/helpers.ts | 68 + flowvault/test/model/options.test.ts | 69 + flowvault/test/model/requests.test.ts | 128 ++ flowvault/test/model/responses.test.ts | 89 ++ flowvault/test/skyflow/skyflow.test.ts | 189 +++ flowvault/test/smoke.test.ts | 232 +++ .../test/validations/validations.test.ts | 260 ++++ flowvault/tsconfig.dts.json | 32 + flowvault/tsconfig.json | 36 + flowvault/tsconfig.test.json | 19 + flowvault/tsup.config.ts | 19 + package-lock.json | 1373 ++++++++++++++++- package.json | 42 +- scripts/bump_version.sh | 70 +- skyvault/.babelrc | 14 + skyvault/LICENSE | 21 + skyvault/README.md | 1325 ++++++++++++++++ skyvault/api-extractor.json | 32 + .../api-report}/skyflow-node.api.md | 573 ++----- skyvault/babel.config.js | 32 + skyvault/jest.config.js | 25 + skyvault/package.json | 60 + .../src}/ _generated_/rest/Client.ts | 0 .../rest/api/errors/BadRequestError.ts | 0 .../rest/api/errors/InternalServerError.ts | 19 + .../rest/api/errors/NotFoundError.ts | 0 .../rest/api/errors/UnauthorizedError.ts | 0 .../ _generated_/rest/api/errors/index.ts | 0 .../src}/ _generated_/rest/api/index.ts | 0 .../rest/api/resources/audit/client/Client.ts | 0 .../rest/api/resources/audit}/client/index.ts | 0 .../AuditServiceListAuditEventsRequest.ts | 0 .../resources/audit/client/requests/index.ts | 0 .../rest/api/resources/audit}/index.ts | 0 ...stAuditEventsRequestFilterOpsActionType.ts | 0 ...EventsRequestFilterOpsContextAccessType.ts | 0 ...tEventsRequestFilterOpsContextActorType.ts | 0 ...itEventsRequestFilterOpsContextAuthMode.ts | 0 ...AuditEventsRequestFilterOpsResourceType.ts | 0 ...iceListAuditEventsRequestSortOpsOrderBy.ts | 0 .../rest/api/resources/audit/types/index.ts | 0 .../resources/authentication/client/Client.ts | 0 .../resources/authentication}/client/index.ts | 0 .../client/requests/V1GetAuthTokenRequest.ts | 0 .../authentication/client/requests/index.ts | 1 + .../api/resources/authentication}/index.ts | 0 .../api/resources/binLookup/client/Client.ts | 0 .../api/resources/binLookup/client/index.ts | 1 + .../client/requests/V1BinListRequest.ts | 0 .../binLookup/client/requests/index.ts | 0 .../rest/api/resources/binLookup/index.ts | 1 + .../rest/api/resources/files/client/Client.ts | 0 .../rest/api/resources/files/client/index.ts | 1 + .../client/requests/DeidentifyAudioRequest.ts | 0 .../requests/DeidentifyDocumentRequest.ts | 0 .../client/requests/DeidentifyFileRequest.ts | 0 .../client/requests/DeidentifyImageRequest.ts | 0 .../client/requests/DeidentifyPdfRequest.ts | 0 .../requests/DeidentifyPresentationRequest.ts | 0 .../requests/DeidentifySpreadsheetRequest.ts | 0 .../DeidentifyStructuredTextRequest.ts | 0 .../client/requests/DeidentifyTextRequest.ts | 0 .../files/client/requests/GetRunRequest.ts | 0 .../client/requests/ReidentifyFileRequest.ts | 0 .../resources/files/client/requests/index.ts | 0 .../rest/api/resources/files}/index.ts | 0 .../files/types/DeidentifyAudioRequestFile.ts | 0 .../DeidentifyAudioRequestFileDataFormat.ts | 0 ...identifyAudioRequestOutputTranscription.ts | 0 .../types/DeidentifyDocumentRequestFile.ts | 0 ...DeidentifyDocumentRequestFileDataFormat.ts | 0 .../files/types/DeidentifyFileRequestFile.ts | 0 .../DeidentifyFileRequestFileDataFormat.ts | 0 .../files/types/DeidentifyImageRequestFile.ts | 0 .../DeidentifyImageRequestFileDataFormat.ts | 0 .../DeidentifyImageRequestMaskingMethod.ts | 0 .../files/types/DeidentifyPdfRequestFile.ts | 0 .../DeidentifyPresentationRequestFile.ts | 0 ...entifyPresentationRequestFileDataFormat.ts | 0 .../types/DeidentifySpreadsheetRequestFile.ts | 0 ...dentifySpreadsheetRequestFileDataFormat.ts | 0 .../DeidentifyStructuredTextRequestFile.ts | 0 ...tifyStructuredTextRequestFileDataFormat.ts | 0 .../files/types/DeidentifyTextRequestFile.ts | 0 .../files/types/ReidentifyFileRequestFile.ts | 0 .../ReidentifyFileRequestFileDataFormat.ts | 0 .../types/ReidentifyFileRequestFormat.ts | 0 .../rest/api/resources/files/types/index.ts | 0 .../api/resources/guardrails/client/Client.ts | 0 .../api/resources/guardrails/client/index.ts | 1 + .../client/requests/CheckGuardrailsRequest.ts | 0 .../guardrails/client/requests/index.ts | 0 .../rest/api/resources/guardrails/index.ts | 1 + .../ _generated_/rest/api/resources/index.ts | 0 .../rest/api/resources/query/client/Client.ts | 0 .../rest/api/resources/query/client/index.ts | 1 + .../requests/QueryServiceExecuteQueryBody.ts | 0 .../resources/query/client/requests/index.ts | 0 .../rest/api/resources/query/index.ts | 1 + .../api/resources/records/client/Client.ts | 0 .../api/resources/records/client/index.ts | 1 + .../requests/FileServiceUploadFileRequest.ts | 0 .../RecordServiceBatchOperationBody.ts | 0 .../RecordServiceBulkDeleteRecordBody.ts | 0 .../RecordServiceBulkGetRecordRequest.ts | 0 .../requests/RecordServiceGetRecordRequest.ts | 0 .../requests/RecordServiceInsertRecordBody.ts | 0 .../requests/RecordServiceUpdateRecordBody.ts | 0 .../client/requests/UploadFileV2Request.ts | 0 .../records/client/requests/index.ts | 0 .../rest/api/resources/records}/index.ts | 0 ...ecordServiceBulkGetRecordRequestOrderBy.ts | 0 ...ordServiceBulkGetRecordRequestRedaction.ts | 0 .../RecordServiceGetRecordRequestRedaction.ts | 0 .../rest/api/resources/records/types/index.ts | 0 .../api/resources/strings/client/Client.ts | 0 .../api/resources/strings/client/index.ts | 1 + .../requests/DeidentifyStringRequest.ts | 0 .../requests/ReidentifyStringRequest.ts | 0 .../strings/client/requests/index.ts | 0 .../rest/api/resources/strings/index.ts | 2 + .../types/ReidentifyStringRequestFormat.ts | 0 .../rest/api/resources/strings/types/index.ts | 0 .../api/resources/tokens/client/Client.ts | 0 .../rest/api/resources/tokens/client/index.ts | 1 + .../client/requests/V1DetokenizePayload.ts | 0 .../client/requests/V1TokenizePayload.ts | 0 .../resources/tokens/client/requests/index.ts | 0 .../rest/api/resources/tokens/index.ts | 1 + .../ _generated_/rest/api/types/AllowRegex.ts | 0 .../api/types/AuditEventAuditResourceType.ts | 0 .../rest/api/types/AuditEventContext.ts | 0 .../rest/api/types/AuditEventData.ts | 0 .../rest/api/types/AuditEventHttpInfo.ts | 0 .../rest/api/types/BatchRecordMethod.ts | 0 .../rest/api/types/CheckGuardrailsResponse.ts | 0 .../CheckGuardrailsResponseValidation.ts | 0 .../rest/api/types/ConfigurationId.ts | 0 .../rest/api/types/ContextAccessType.ts | 0 .../rest/api/types/ContextAuthMode.ts | 0 .../rest/api/types/DeidentifyFileOutput.ts | 0 .../DeidentifyFileOutputProcessedFileType.ts | 0 .../rest/api/types/DeidentifyFileResponse.ts | 0 .../api/types/DeidentifyStatusResponse.ts | 0 .../DeidentifyStatusResponseOutputType.ts | 0 .../types/DeidentifyStatusResponseStatus.ts | 0 .../api/types/DeidentifyStringResponse.ts | 0 .../rest/api/types/DetectedEntity.ts | 0 .../DetokenizeRecordResponseValueType.ts | 0 .../rest/api/types/EntityLocation.ts | 0 .../ _generated_/rest/api/types/EntityType.ts | 0 .../rest/api/types/EntityTypes.ts | 0 .../rest/api/types/ErrorResponse.ts | 9 + .../rest/api/types/ErrorResponseError.ts | 0 .../rest/api/types/GooglerpcStatus.ts | 11 + .../rest/api/types/ProtobufAny.ts | 7 + .../rest/api/types/RedactionEnumRedaction.ts | 0 .../rest/api/types/ReidentifyFileResponse.ts | 0 .../api/types/ReidentifyFileResponseOutput.ts | 0 .../types/ReidentifyFileResponseOutputType.ts | 0 .../api/types/ReidentifyFileResponseStatus.ts | 0 .../api/types/ReidentifyStringResponse.ts | 0 .../rest/api/types/RequestActionType.ts | 0 .../ _generated_/rest/api/types/ResourceId.ts | 0 .../rest/api/types/RestrictRegex.ts | 0 .../ _generated_/rest/api/types/TokenType.ts | 0 .../rest/api/types/TokenTypeDefault.ts | 0 .../rest/api/types/TokenTypeWithoutVault.ts | 0 .../api/types/TokenTypeWithoutVaultDefault.ts | 0 .../rest/api/types/Transformations.ts | 0 .../api/types/TransformationsShiftDates.ts | 0 ...ransformationsShiftDatesEntityTypesItem.ts | 0 .../rest/api/types/UploadFileV2Response.ts | 0 .../src}/ _generated_/rest/api/types/Uuid.ts | 0 .../rest/api/types/V1AuditAfterOptions.ts | 0 .../rest/api/types/V1AuditEventResponse.ts | 0 .../rest/api/types/V1AuditResponse.ts | 0 .../rest/api/types/V1AuditResponseEvent.ts | 0 .../api/types/V1AuditResponseEventRequest.ts | 0 .../api/types/V1BatchOperationResponse.ts | 0 .../rest/api/types/V1BatchRecord.ts | 0 .../rest/api/types/V1BinListResponse.ts | 0 .../api/types/V1BulkDeleteRecordResponse.ts | 0 .../rest/api/types/V1BulkGetRecordResponse.ts | 0 .../ _generated_/rest/api/types/V1Byot.ts | 0 .../ _generated_/rest/api/types/V1Card.ts | 0 .../rest/api/types/V1DeleteFileResponse.ts | 0 .../rest/api/types/V1DeleteRecordResponse.ts | 0 .../api/types/V1DetokenizeRecordRequest.ts | 0 .../api/types/V1DetokenizeRecordResponse.ts | 0 .../rest/api/types/V1DetokenizeResponse.ts | 0 .../rest/api/types/V1FieldRecords.ts | 0 .../rest/api/types/V1FileAvScanStatus.ts | 0 .../rest/api/types/V1GetAuthTokenResponse.ts | 10 + .../api/types/V1GetFileScanStatusResponse.ts | 0 .../rest/api/types/V1GetQueryResponse.ts | 0 .../rest/api/types/V1InsertRecordResponse.ts | 0 .../rest/api/types/V1MemberType.ts | 0 .../rest/api/types/V1RecordMetaProperties.ts | 0 .../rest/api/types/V1TokenizeRecordRequest.ts | 0 .../api/types/V1TokenizeRecordResponse.ts | 0 .../rest/api/types/V1TokenizeResponse.ts | 0 .../rest/api/types/V1UpdateRecordResponse.ts | 0 .../rest/api/types/V1VaultFieldMapping.ts | 0 .../rest/api/types/V1VaultSchemaConfig.ts | 0 .../ _generated_/rest/api/types/VaultId.ts | 0 .../src}/ _generated_/rest/api/types/index.ts | 0 .../ _generated_/rest/core/auth/BasicAuth.ts | 31 + .../rest/core/auth/BearerToken.ts | 15 + .../src/ _generated_/rest/core/auth/index.ts | 2 + .../rest/core/fetcher/APIResponse.ts | 23 + .../ _generated_/rest/core/fetcher/Fetcher.ts | 151 ++ .../rest/core/fetcher/HttpResponsePromise.ts | 116 ++ .../rest/core/fetcher/RawResponse.ts | 59 + .../rest/core/fetcher/Supplier.ts | 11 + .../rest/core/fetcher/createRequestUrl.ts | 10 + .../rest/core/fetcher/getFetchFn.ts | 25 + .../rest/core/fetcher/getHeader.ts | 8 + .../rest/core/fetcher/getRequestBody.ts | 16 + .../rest/core/fetcher/getResponseBody.ts | 34 + .../ _generated_/rest/core/fetcher/index.ts | 8 + .../rest/core/fetcher/makeRequest.ts | 44 + .../rest/core/fetcher/requestWithRetries.ts | 33 + .../ _generated_/rest/core/fetcher/signals.ts | 38 + .../Node18UniversalStreamWrapper.ts | 0 .../stream-wrappers/NodePre18StreamWrapper.ts | 107 ++ .../stream-wrappers/UndiciStreamWrapper.ts | 0 .../stream-wrappers/chooseStreamWrapper.ts | 34 + .../core/form-data-utils/FormDataWrapper.ts | 0 .../form-data-utils/encodeAsFormParameter.ts | 0 .../rest/core/form-data-utils/index.ts | 0 .../src}/ _generated_/rest/core/index.ts | 0 skyvault/src/ _generated_/rest/core/json.ts | 27 + .../ _generated_/rest/core/runtime/index.ts | 1 + .../ _generated_/rest/core/runtime/runtime.ts | 131 ++ .../src}/ _generated_/rest/environments.ts | 0 .../ _generated_/rest/errors/SkyflowError.ts | 55 + .../rest/errors/SkyflowTimeoutError.ts | 10 + .../src/ _generated_/rest/errors/index.ts | 2 + skyvault/src/ _generated_/rest/index.ts | 4 + skyvault/src/ _generated_/rest/version.ts | 1 + {src => skyvault/src}/index.ts | 20 +- .../management/config/management/index.ts | 0 .../management/controller/policy/index.ts | 0 .../src}/management/controller/role/index.ts | 0 .../src}/management/controller/vault/index.ts | 0 {src => skyvault/src}/management/index.ts | 0 .../model/request/create-vault/index.ts | 0 .../model/request/delete-policy/index.ts | 0 .../model/response/create-vault/index.ts | 0 .../model/response/delete-policy/index.ts | 0 .../src}/management/skyflow/index.ts | 0 .../src}/management/types/index.ts | 2 +- .../src}/utils/validations/index.ts | 380 +---- {src => skyvault/src}/vault/client/index.ts | 12 +- .../src}/vault/controller/audit/index.ts | 0 .../src}/vault/controller/binlookup/index.ts | 0 .../vault/controller/connections/index.ts | 8 +- .../src}/vault/controller/detect/index.ts | 8 +- .../src}/vault/controller/vault/index.ts | 8 +- .../deidentify-file/bleep-audio/index.ts | 0 .../model/options/deidentify-file/index.ts | 2 +- .../model/options/deidentify-text/index.ts | 2 +- .../deidentify-text/token-format/index.ts | 2 +- .../deidentify-text/transformations/index.ts | 2 +- .../vault/model/options/detokenize/index.ts | 9 +- .../vault/model/options/fileUpload/index.ts | 5 +- .../src}/vault/model/options/get/index.ts | 9 +- .../src}/vault/model/options/insert/index.ts | 7 +- .../model/options/reidentify-text/index.ts | 2 +- .../src}/vault/model/options/update/index.ts | 6 +- .../model/request/deidentify-file/index.ts | 0 .../model/request/deidentify-text/index.ts | 0 .../src}/vault/model/request/delete/index.ts | 4 +- .../vault/model/request/detokenize/index.ts | 4 +- .../vault/model/request/file-upload/index.ts | 8 +- .../vault/model/request/get-column/index.ts | 0 .../model/request/get-detect-run/index.ts | 0 .../src}/vault/model/request/get/index.ts | 6 +- .../src}/vault/model/request/inkove/index.ts | 6 +- .../src}/vault/model/request/insert/index.ts | 4 +- .../src}/vault/model/request/query/index.ts | 4 +- .../model/request/reidentify-text/index.ts | 0 .../vault/model/request/tokenize/index.ts | 4 +- .../src}/vault/model/request/update/index.ts | 4 +- .../model/response/deidentify-file/index.ts | 2 +- .../model/response/deidentify-text/index.ts | 2 +- .../src}/vault/model/response/delete/index.ts | 6 +- .../vault/model/response/detokenize/index.ts | 6 +- .../vault/model/response/file-upload/index.ts | 6 +- .../src}/vault/model/response/get/index.ts | 6 +- .../src}/vault/model/response/insert/index.ts | 8 +- .../vault/model/response/invoke/invoke.ts | 6 +- .../src}/vault/model/response/query/index.ts | 6 +- .../model/response/reidentify-text/index.ts | 0 .../vault/model/response/tokenize/index.ts | 6 +- .../src}/vault/model/response/update/index.ts | 8 +- skyvault/src/vault/skyflow/index.ts | 201 +++ {src => skyvault/src}/vault/types/index.ts | 88 +- skyvault/src/version.ts | 4 + skyvault/test/demo-credentials/empty.json | 0 .../test/demo-credentials/invalidJson.json | 1 + .../test/demo-credentials/noClientId.json | 3 + skyvault/test/demo-credentials/noKeyId.json | 4 + .../test/demo-credentials/noPrivateKey.json | 1 + .../test/demo-credentials/noTokenURI.json | 5 + skyvault/test/demo-credentials/valid.json | 7 + {test => skyvault/test}/mockData/pii.pdf | Bin .../test}/utils/validations.test.js | 26 +- .../test}/vault/client/client.test.js | 15 +- .../test}/vault/controller/connection.test.js | 79 +- .../controller/deprecated-detokenize.test.js | 46 +- .../controller/deprecated-insert.test.js | 45 +- .../test}/vault/controller/detect.test.js | 83 +- .../test}/vault/controller/vault.test.js | 100 +- .../test}/vault/model/deprecated.test.js | 29 +- .../test}/vault/skyflow/skyflow.test.js | 0 skyvault/tsconfig.dts.json | 25 + skyvault/tsconfig.json | 30 + skyvault/tsup.config.ts | 19 + src/error/messages/index.ts | 252 --- src/vault/config/vault/index.ts | 12 - src/vault/skyflow/index.ts | 274 ---- tsconfig.json | 30 +- 600 files changed, 18045 insertions(+), 2013 deletions(-) create mode 100644 .github/workflows/flowvault-beta-release.yml create mode 100644 .github/workflows/flowvault-release.yml delete mode 100644 api-extractor.json create mode 100644 common/.babelrc create mode 100644 common/babel.config.js create mode 100644 common/jest.config.js create mode 100644 common/package.json create mode 100644 common/src/_generated_/rest/Client.ts rename {src/ _generated_/rest/api/resources/audit => common/src/_generated_/rest/api}/client/index.ts (100%) create mode 100644 common/src/_generated_/rest/api/client/requests/PatchV2VaultsIdRequest.ts create mode 100644 common/src/_generated_/rest/api/client/requests/index.ts create mode 100644 common/src/_generated_/rest/api/errors/BadRequestError.ts create mode 100644 common/src/_generated_/rest/api/errors/NotFoundError.ts create mode 100644 common/src/_generated_/rest/api/errors/UnauthorizedError.ts create mode 100644 common/src/_generated_/rest/api/errors/index.ts create mode 100644 common/src/_generated_/rest/api/index.ts create mode 100644 common/src/_generated_/rest/api/resources/authentication/client/Client.ts rename {src/ _generated_ => common/src/_generated_}/rest/api/resources/authentication/client/index.ts (100%) create mode 100644 common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts rename {src/ _generated_ => common/src/_generated_}/rest/api/resources/authentication/client/requests/index.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/api/resources/authentication/index.ts (100%) create mode 100644 common/src/_generated_/rest/api/resources/index.ts rename {src/ _generated_ => common/src/_generated_}/rest/api/types/GooglerpcStatus.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/api/types/ProtobufAny.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/api/types/V1GetAuthTokenResponse.ts (100%) create mode 100644 common/src/_generated_/rest/api/types/index.ts rename {src/ _generated_ => common/src/_generated_}/rest/core/auth/BasicAuth.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/auth/BearerToken.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/auth/index.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/APIResponse.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/Fetcher.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/HttpResponsePromise.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/RawResponse.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/Supplier.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/createRequestUrl.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/getFetchFn.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/getHeader.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/getRequestBody.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/getResponseBody.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/index.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/makeRequest.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/requestWithRetries.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/signals.ts (100%) create mode 100644 common/src/_generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts (100%) create mode 100644 common/src/_generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts rename {src/ _generated_ => common/src/_generated_}/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts (100%) create mode 100644 common/src/_generated_/rest/core/index.ts rename {src/ _generated_ => common/src/_generated_}/rest/core/json.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/runtime/index.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/core/runtime/runtime.ts (100%) create mode 100644 common/src/_generated_/rest/environments.ts rename {src/ _generated_ => common/src/_generated_}/rest/errors/SkyflowError.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/errors/SkyflowTimeoutError.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/errors/index.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/index.ts (100%) rename {src/ _generated_ => common/src/_generated_}/rest/version.ts (100%) create mode 100644 common/src/base/options.ts create mode 100644 common/src/base/request.ts create mode 100644 common/src/base/response.ts rename {src/vault => common/src}/config/connection/index.ts (100%) rename {src/vault => common/src}/config/credentials/index.ts (100%) create mode 100644 common/src/config/vault/index.ts rename {src => common/src}/error/codes/index.ts (95%) rename {src => common/src}/error/index.ts (100%) create mode 100644 common/src/error/messages/index.ts create mode 100644 common/src/identity.ts create mode 100644 common/src/index.ts rename {src => common/src}/service-account/client/index.ts (69%) rename {src => common/src}/service-account/index.ts (99%) create mode 100644 common/src/skyflow/base.ts create mode 100644 common/src/types/index.ts rename {src => common/src}/utils/index.ts (97%) rename {src => common/src}/utils/jwt-utils/index.ts (100%) rename {src => common/src}/utils/logs/index.ts (100%) create mode 100644 common/src/validations/index.ts rename {test => common/test}/demo-credentials/empty.json (100%) rename {test => common/test}/demo-credentials/invalidJson.json (100%) rename {test => common/test}/demo-credentials/noClientId.json (100%) rename {test => common/test}/demo-credentials/noKeyId.json (100%) rename {test => common/test}/demo-credentials/noPrivateKey.json (100%) rename {test => common/test}/demo-credentials/noTokenURI.json (100%) rename {test => common/test}/demo-credentials/valid.json (100%) rename {test => common/test}/error/skyflow-error.test.js (100%) rename {test => common/test}/service-account/client.test.js (78%) rename {test => common/test}/service-account/token.test.js (100%) rename {test/vault/utils/jwt-utils => common/test/utils}/jwt.test.js (93%) rename {test/vault => common/test}/utils/utils.test.js (96%) create mode 100644 common/tsconfig.json create mode 100644 docs/sdk-split-plan.md create mode 100644 flowvault/.gitignore create mode 100644 flowvault/LICENSE create mode 100644 flowvault/README.md create mode 100644 flowvault/_generated_/rest/src/Client.ts rename {src/ _generated_/rest/api/resources/binLookup => flowvault/_generated_/rest/src/api}/client/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/client/requests/PatchV2VaultsIdRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/client/requests/index.ts create mode 100644 flowvault/_generated_/rest/src/api/errors/BadRequestError.ts create mode 100644 flowvault/_generated_/rest/src/api/errors/ForbiddenError.ts rename {src/ _generated_/rest => flowvault/_generated_/rest/src}/api/errors/InternalServerError.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/errors/NotFoundError.ts create mode 100644 flowvault/_generated_/rest/src/api/errors/TooManyRequestsError.ts create mode 100644 flowvault/_generated_/rest/src/api/errors/UnauthorizedError.ts create mode 100644 flowvault/_generated_/rest/src/api/errors/index.ts create mode 100644 flowvault/_generated_/rest/src/api/index.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/files/client/Client.ts rename {src/ _generated_/rest => flowvault/_generated_/rest/src}/api/resources/files/client/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/resources/files/client/requests/FileDeleteRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/files/client/requests/FileUploadRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/files/client/requests/index.ts rename {src/ _generated_/rest/api/resources/binLookup => flowvault/_generated_/rest/src/api/resources/files}/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/resources/index.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/query/client/Client.ts rename {src/ _generated_/rest/api/resources/guardrails => flowvault/_generated_/rest/src/api/resources/query}/client/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/resources/query/client/requests/ExecuteQueryRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/query/client/requests/index.ts rename {src/ _generated_/rest/api/resources/guardrails => flowvault/_generated_/rest/src/api/resources/query}/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/resources/records/client/Client.ts rename {src/ _generated_/rest/api/resources/query => flowvault/_generated_/rest/src/api/resources/records}/client/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/resources/records/client/requests/DeleteRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/records/client/requests/GetRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/records/client/requests/InsertRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/records/client/requests/UpdateRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/records/client/requests/index.ts rename {src/ _generated_/rest/api/resources/audit => flowvault/_generated_/rest/src/api/resources/records}/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/resources/records/types/UpdateRequestUpdateType.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/records/types/index.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/tokens/client/Client.ts rename {src/ _generated_/rest/api/resources/records => flowvault/_generated_/rest/src/api/resources/tokens}/client/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/resources/tokens/client/requests/DetokenizeRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/tokens/client/requests/GetTokensFromValuesRequest.ts create mode 100644 flowvault/_generated_/rest/src/api/resources/tokens/client/requests/index.ts rename {src/ _generated_/rest/api/resources/query => flowvault/_generated_/rest/src/api/resources/tokens}/index.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/types/ColumnRedactions.ts create mode 100644 flowvault/_generated_/rest/src/api/types/DeleteResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/DeleteResponseObject.ts create mode 100644 flowvault/_generated_/rest/src/api/types/DetokenizeResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/DetokenizeResponseObject.ts rename {src/ _generated_/rest => flowvault/_generated_/rest/src}/api/types/ErrorResponse.ts (100%) create mode 100644 flowvault/_generated_/rest/src/api/types/ErrorResponseError.ts create mode 100644 flowvault/_generated_/rest/src/api/types/ExecuteQueryRecordResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/ExecuteQueryResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/ExecuteQueryResponseMetadata.ts create mode 100644 flowvault/_generated_/rest/src/api/types/FileDeleteRecord.ts create mode 100644 flowvault/_generated_/rest/src/api/types/FileDeleteResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/FileDeleteResponseObject.ts create mode 100644 flowvault/_generated_/rest/src/api/types/FileUploadColumn.ts create mode 100644 flowvault/_generated_/rest/src/api/types/FileUploadRecord.ts create mode 100644 flowvault/_generated_/rest/src/api/types/FileUploadResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/FileUploadResponseObject.ts create mode 100644 flowvault/_generated_/rest/src/api/types/GetRequestData.ts create mode 100644 flowvault/_generated_/rest/src/api/types/GetResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/GetTokensFromValuesRequestObject.ts create mode 100644 flowvault/_generated_/rest/src/api/types/GetTokensFromValuesResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/GoogleProtobufValue.ts create mode 100644 flowvault/_generated_/rest/src/api/types/HttpCode.ts create mode 100644 flowvault/_generated_/rest/src/api/types/InsertRecordData.ts create mode 100644 flowvault/_generated_/rest/src/api/types/InsertResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/RecordResponseObject.ts create mode 100644 flowvault/_generated_/rest/src/api/types/TokenGroupRedactions.ts create mode 100644 flowvault/_generated_/rest/src/api/types/TokenizeResponseObject.ts create mode 100644 flowvault/_generated_/rest/src/api/types/UniqueValue.ts create mode 100644 flowvault/_generated_/rest/src/api/types/UpdateRecordData.ts create mode 100644 flowvault/_generated_/rest/src/api/types/UpdateRecordDataUpdateType.ts create mode 100644 flowvault/_generated_/rest/src/api/types/UpdateResponse.ts create mode 100644 flowvault/_generated_/rest/src/api/types/Upsert.ts create mode 100644 flowvault/_generated_/rest/src/api/types/UpsertUpdateType.ts create mode 100644 flowvault/_generated_/rest/src/api/types/index.ts create mode 100644 flowvault/_generated_/rest/src/core/auth/BasicAuth.ts create mode 100644 flowvault/_generated_/rest/src/core/auth/BearerToken.ts create mode 100644 flowvault/_generated_/rest/src/core/auth/index.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/APIResponse.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/Fetcher.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/HttpResponsePromise.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/RawResponse.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/Supplier.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/createRequestUrl.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/getFetchFn.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/getHeader.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/getRequestBody.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/getResponseBody.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/index.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/makeRequest.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/requestWithRetries.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/signals.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts create mode 100644 flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/chooseStreamWrapper.ts create mode 100644 flowvault/_generated_/rest/src/core/index.ts create mode 100644 flowvault/_generated_/rest/src/core/json.ts create mode 100644 flowvault/_generated_/rest/src/core/runtime/index.ts create mode 100644 flowvault/_generated_/rest/src/core/runtime/runtime.ts create mode 100644 flowvault/_generated_/rest/src/environments.ts create mode 100644 flowvault/_generated_/rest/src/errors/SkyflowError.ts create mode 100644 flowvault/_generated_/rest/src/errors/SkyflowTimeoutError.ts create mode 100644 flowvault/_generated_/rest/src/errors/index.ts create mode 100644 flowvault/_generated_/rest/src/index.ts create mode 100644 flowvault/_generated_/rest/src/version.ts create mode 100644 flowvault/api-extractor.json create mode 100644 flowvault/api-report/skyflow-flowvault-node.api.md create mode 100644 flowvault/jest.config.js create mode 100644 flowvault/package.json create mode 100644 flowvault/samples/README.md create mode 100644 flowvault/samples/service-account/scoped-token-generation-example.ts create mode 100644 flowvault/samples/service-account/signed-token-generation-example.ts create mode 100644 flowvault/samples/service-account/token-generation-example.ts create mode 100644 flowvault/samples/vault-api/custom-header-example.ts create mode 100644 flowvault/samples/vault-api/delete-files.ts create mode 100644 flowvault/samples/vault-api/delete-records.ts create mode 100644 flowvault/samples/vault-api/detokenize-records.ts create mode 100644 flowvault/samples/vault-api/get-records.ts create mode 100644 flowvault/samples/vault-api/get-tokens.ts create mode 100644 flowvault/samples/vault-api/insert-records.ts create mode 100644 flowvault/samples/vault-api/query-records.ts create mode 100644 flowvault/samples/vault-api/timeout-and-retry-config-example.ts create mode 100644 flowvault/samples/vault-api/update-record.ts create mode 100644 flowvault/samples/vault-api/upload-files.ts create mode 100644 flowvault/src/client/index.ts create mode 100644 flowvault/src/controller/vault/index.ts create mode 100644 flowvault/src/enums/custom-header-key.ts create mode 100644 flowvault/src/index.ts create mode 100644 flowvault/src/model/options/base.ts create mode 100644 flowvault/src/model/options/delete-files/index.ts create mode 100644 flowvault/src/model/options/delete/index.ts create mode 100644 flowvault/src/model/options/detokenize/index.ts create mode 100644 flowvault/src/model/options/get-tokens/index.ts create mode 100644 flowvault/src/model/options/get/index.ts create mode 100644 flowvault/src/model/options/insert/index.ts create mode 100644 flowvault/src/model/options/query/index.ts create mode 100644 flowvault/src/model/options/update/index.ts create mode 100644 flowvault/src/model/options/upload-files/index.ts create mode 100644 flowvault/src/model/request-context/index.ts create mode 100644 flowvault/src/model/request/delete-files/index.ts create mode 100644 flowvault/src/model/request/delete/index.ts create mode 100644 flowvault/src/model/request/detokenize/index.ts create mode 100644 flowvault/src/model/request/get-tokens/index.ts create mode 100644 flowvault/src/model/request/get/index.ts create mode 100644 flowvault/src/model/request/insert/index.ts create mode 100644 flowvault/src/model/request/query/index.ts create mode 100644 flowvault/src/model/request/update/index.ts create mode 100644 flowvault/src/model/request/upload-files/index.ts create mode 100644 flowvault/src/model/response/delete-files/index.ts create mode 100644 flowvault/src/model/response/delete/index.ts create mode 100644 flowvault/src/model/response/detokenize/index.ts create mode 100644 flowvault/src/model/response/get-tokens/index.ts create mode 100644 flowvault/src/model/response/get/index.ts create mode 100644 flowvault/src/model/response/insert/index.ts create mode 100644 flowvault/src/model/response/query/index.ts create mode 100644 flowvault/src/model/response/update/index.ts create mode 100644 flowvault/src/model/response/upload-files/index.ts create mode 100644 flowvault/src/skyflow/index.ts create mode 100644 flowvault/src/types/index.ts create mode 100644 flowvault/src/validations/index.ts create mode 100644 flowvault/src/version.ts create mode 100644 flowvault/test/client/client.test.ts create mode 100644 flowvault/test/controller/delete-files.test.ts create mode 100644 flowvault/test/controller/delete.test.ts create mode 100644 flowvault/test/controller/detokenize.test.ts create mode 100644 flowvault/test/controller/error-records.test.ts create mode 100644 flowvault/test/controller/get-tokens.test.ts create mode 100644 flowvault/test/controller/get.test.ts create mode 100644 flowvault/test/controller/insert.test.ts create mode 100644 flowvault/test/controller/query.test.ts create mode 100644 flowvault/test/controller/request-options.test.ts create mode 100644 flowvault/test/controller/update.test.ts create mode 100644 flowvault/test/controller/upload-files.test.ts create mode 100644 flowvault/test/helpers.ts create mode 100644 flowvault/test/model/options.test.ts create mode 100644 flowvault/test/model/requests.test.ts create mode 100644 flowvault/test/model/responses.test.ts create mode 100644 flowvault/test/skyflow/skyflow.test.ts create mode 100644 flowvault/test/smoke.test.ts create mode 100644 flowvault/test/validations/validations.test.ts create mode 100644 flowvault/tsconfig.dts.json create mode 100644 flowvault/tsconfig.json create mode 100644 flowvault/tsconfig.test.json create mode 100644 flowvault/tsup.config.ts create mode 100644 skyvault/.babelrc create mode 100644 skyvault/LICENSE create mode 100644 skyvault/README.md create mode 100644 skyvault/api-extractor.json rename {api-report => skyvault/api-report}/skyflow-node.api.md (59%) create mode 100644 skyvault/babel.config.js create mode 100644 skyvault/jest.config.js create mode 100644 skyvault/package.json rename {src => skyvault/src}/ _generated_/rest/Client.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/errors/BadRequestError.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/errors/InternalServerError.ts rename {src => skyvault/src}/ _generated_/rest/api/errors/NotFoundError.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/errors/UnauthorizedError.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/errors/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/client/Client.ts (100%) rename {src/ _generated_/rest/api/resources/strings => skyvault/src/ _generated_/rest/api/resources/audit}/client/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/client/requests/AuditServiceListAuditEventsRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/client/requests/index.ts (100%) rename {src/ _generated_/rest/api/resources/files => skyvault/src/ _generated_/rest/api/resources/audit}/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsActionType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAccessType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextActorType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAuthMode.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsResourceType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestSortOpsOrderBy.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/audit/types/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/authentication/client/Client.ts (100%) rename {src/ _generated_/rest/api/resources/tokens => skyvault/src/ _generated_/rest/api/resources/authentication}/client/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/authentication/client/requests/index.ts rename {src/ _generated_/rest/api/resources/tokens => skyvault/src/ _generated_/rest/api/resources/authentication}/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/binLookup/client/Client.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/binLookup/client/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/binLookup/client/requests/V1BinListRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/binLookup/client/requests/index.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/binLookup/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/Client.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/files/client/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyAudioRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyDocumentRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyFileRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyImageRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyPdfRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyPresentationRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifySpreadsheetRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyStructuredTextRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/DeidentifyTextRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/GetRunRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/ReidentifyFileRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/client/requests/index.ts (100%) rename {src/ _generated_/rest/api/resources/records => skyvault/src/ _generated_/rest/api/resources/files}/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestOutputTranscription.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestMaskingMethod.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyPdfRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/DeidentifyTextRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFile.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFileDataFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/files/types/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/guardrails/client/Client.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/guardrails/client/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/guardrails/client/requests/CheckGuardrailsRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/guardrails/client/requests/index.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/guardrails/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/query/client/Client.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/query/client/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/query/client/requests/QueryServiceExecuteQueryBody.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/query/client/requests/index.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/query/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/Client.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/records/client/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/FileServiceUploadFileRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/RecordServiceBatchOperationBody.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkDeleteRecordBody.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkGetRecordRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/RecordServiceGetRecordRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/RecordServiceInsertRecordBody.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/RecordServiceUpdateRecordBody.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/UploadFileV2Request.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/client/requests/index.ts (100%) rename {src/ _generated_/rest/api/resources/strings => skyvault/src/ _generated_/rest/api/resources/records}/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestOrderBy.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestRedaction.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/types/RecordServiceGetRecordRequestRedaction.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/records/types/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/strings/client/Client.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/strings/client/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/strings/client/requests/DeidentifyStringRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/strings/client/requests/ReidentifyStringRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/strings/client/requests/index.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/strings/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/strings/types/ReidentifyStringRequestFormat.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/strings/types/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/tokens/client/Client.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/tokens/client/index.ts rename {src => skyvault/src}/ _generated_/rest/api/resources/tokens/client/requests/V1DetokenizePayload.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/tokens/client/requests/V1TokenizePayload.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/resources/tokens/client/requests/index.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/resources/tokens/index.ts rename {src => skyvault/src}/ _generated_/rest/api/types/AllowRegex.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/AuditEventAuditResourceType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/AuditEventContext.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/AuditEventData.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/AuditEventHttpInfo.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/BatchRecordMethod.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/CheckGuardrailsResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/CheckGuardrailsResponseValidation.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ConfigurationId.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ContextAccessType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ContextAuthMode.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DeidentifyFileOutput.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DeidentifyFileOutputProcessedFileType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DeidentifyFileResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DeidentifyStatusResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DeidentifyStatusResponseOutputType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DeidentifyStatusResponseStatus.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DeidentifyStringResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DetectedEntity.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/DetokenizeRecordResponseValueType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/EntityLocation.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/EntityType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/EntityTypes.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/types/ErrorResponse.ts rename {src => skyvault/src}/ _generated_/rest/api/types/ErrorResponseError.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/types/GooglerpcStatus.ts create mode 100644 skyvault/src/ _generated_/rest/api/types/ProtobufAny.ts rename {src => skyvault/src}/ _generated_/rest/api/types/RedactionEnumRedaction.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ReidentifyFileResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ReidentifyFileResponseOutput.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ReidentifyFileResponseOutputType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ReidentifyFileResponseStatus.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ReidentifyStringResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/RequestActionType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/ResourceId.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/RestrictRegex.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/TokenType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/TokenTypeDefault.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/TokenTypeWithoutVault.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/TokenTypeWithoutVaultDefault.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/Transformations.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/TransformationsShiftDates.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/TransformationsShiftDatesEntityTypesItem.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/UploadFileV2Response.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/Uuid.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1AuditAfterOptions.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1AuditEventResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1AuditResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1AuditResponseEvent.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1AuditResponseEventRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1BatchOperationResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1BatchRecord.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1BinListResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1BulkDeleteRecordResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1BulkGetRecordResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1Byot.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1Card.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1DeleteFileResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1DeleteRecordResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1DetokenizeRecordRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1DetokenizeRecordResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1DetokenizeResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1FieldRecords.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1FileAvScanStatus.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/api/types/V1GetAuthTokenResponse.ts rename {src => skyvault/src}/ _generated_/rest/api/types/V1GetFileScanStatusResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1GetQueryResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1InsertRecordResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1MemberType.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1RecordMetaProperties.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1TokenizeRecordRequest.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1TokenizeRecordResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1TokenizeResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1UpdateRecordResponse.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1VaultFieldMapping.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/V1VaultSchemaConfig.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/VaultId.ts (100%) rename {src => skyvault/src}/ _generated_/rest/api/types/index.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/core/auth/BasicAuth.ts create mode 100644 skyvault/src/ _generated_/rest/core/auth/BearerToken.ts create mode 100644 skyvault/src/ _generated_/rest/core/auth/index.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/APIResponse.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/Fetcher.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/HttpResponsePromise.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/RawResponse.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/Supplier.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/createRequestUrl.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/getFetchFn.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/getHeader.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/getRequestBody.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/getResponseBody.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/index.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/makeRequest.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/requestWithRetries.ts create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/signals.ts rename {src => skyvault/src}/ _generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts rename {src => skyvault/src}/ _generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts rename {src => skyvault/src}/ _generated_/rest/core/form-data-utils/FormDataWrapper.ts (100%) rename {src => skyvault/src}/ _generated_/rest/core/form-data-utils/encodeAsFormParameter.ts (100%) rename {src => skyvault/src}/ _generated_/rest/core/form-data-utils/index.ts (100%) rename {src => skyvault/src}/ _generated_/rest/core/index.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/core/json.ts create mode 100644 skyvault/src/ _generated_/rest/core/runtime/index.ts create mode 100644 skyvault/src/ _generated_/rest/core/runtime/runtime.ts rename {src => skyvault/src}/ _generated_/rest/environments.ts (100%) create mode 100644 skyvault/src/ _generated_/rest/errors/SkyflowError.ts create mode 100644 skyvault/src/ _generated_/rest/errors/SkyflowTimeoutError.ts create mode 100644 skyvault/src/ _generated_/rest/errors/index.ts create mode 100644 skyvault/src/ _generated_/rest/index.ts create mode 100644 skyvault/src/ _generated_/rest/version.ts rename {src => skyvault/src}/index.ts (91%) rename {src => skyvault/src}/management/config/management/index.ts (100%) rename {src => skyvault/src}/management/controller/policy/index.ts (100%) rename {src => skyvault/src}/management/controller/role/index.ts (100%) rename {src => skyvault/src}/management/controller/vault/index.ts (100%) rename {src => skyvault/src}/management/index.ts (100%) rename {src => skyvault/src}/management/model/request/create-vault/index.ts (100%) rename {src => skyvault/src}/management/model/request/delete-policy/index.ts (100%) rename {src => skyvault/src}/management/model/response/create-vault/index.ts (100%) rename {src => skyvault/src}/management/model/response/delete-policy/index.ts (100%) rename {src => skyvault/src}/management/skyflow/index.ts (100%) rename {src => skyvault/src}/management/types/index.ts (78%) rename {src => skyvault/src}/utils/validations/index.ts (69%) rename {src => skyvault/src}/vault/client/index.ts (98%) rename {src => skyvault/src}/vault/controller/audit/index.ts (100%) rename {src => skyvault/src}/vault/controller/binlookup/index.ts (100%) rename {src => skyvault/src}/vault/controller/connections/index.ts (98%) rename {src => skyvault/src}/vault/controller/detect/index.ts (99%) rename {src => skyvault/src}/vault/controller/vault/index.ts (99%) rename {src => skyvault/src}/vault/model/options/deidentify-file/bleep-audio/index.ts (100%) rename {src => skyvault/src}/vault/model/options/deidentify-file/index.ts (99%) rename {src => skyvault/src}/vault/model/options/deidentify-text/index.ts (96%) rename {src => skyvault/src}/vault/model/options/deidentify-text/token-format/index.ts (94%) rename {src => skyvault/src}/vault/model/options/deidentify-text/transformations/index.ts (89%) rename {src => skyvault/src}/vault/model/options/detokenize/index.ts (82%) rename {src => skyvault/src}/vault/model/options/fileUpload/index.ts (89%) rename {src => skyvault/src}/vault/model/options/get/index.ts (93%) rename {src => skyvault/src}/vault/model/options/insert/index.ts (89%) rename {src => skyvault/src}/vault/model/options/reidentify-text/index.ts (94%) rename {src => skyvault/src}/vault/model/options/update/index.ts (83%) rename {src => skyvault/src}/vault/model/request/deidentify-file/index.ts (100%) rename {src => skyvault/src}/vault/model/request/deidentify-text/index.ts (100%) rename {src => skyvault/src}/vault/model/request/delete/index.ts (84%) rename {src => skyvault/src}/vault/model/request/detokenize/index.ts (77%) rename {src => skyvault/src}/vault/model/request/file-upload/index.ts (87%) rename {src => skyvault/src}/vault/model/request/get-column/index.ts (100%) rename {src => skyvault/src}/vault/model/request/get-detect-run/index.ts (100%) rename {src => skyvault/src}/vault/model/request/get/index.ts (80%) rename {src => skyvault/src}/vault/model/request/inkove/index.ts (77%) rename {src => skyvault/src}/vault/model/request/insert/index.ts (85%) rename {src => skyvault/src}/vault/model/request/query/index.ts (73%) rename {src => skyvault/src}/vault/model/request/reidentify-text/index.ts (100%) rename {src => skyvault/src}/vault/model/request/tokenize/index.ts (80%) rename {src => skyvault/src}/vault/model/request/update/index.ts (85%) rename {src => skyvault/src}/vault/model/response/deidentify-file/index.ts (96%) rename {src => skyvault/src}/vault/model/response/deidentify-text/index.ts (95%) rename {src => skyvault/src}/vault/model/response/delete/index.ts (73%) rename {src => skyvault/src}/vault/model/response/detokenize/index.ts (74%) rename {src => skyvault/src}/vault/model/response/file-upload/index.ts (64%) rename {src => skyvault/src}/vault/model/response/get/index.ts (70%) rename {src => skyvault/src}/vault/model/response/insert/index.ts (70%) rename {src => skyvault/src}/vault/model/response/invoke/invoke.ts (68%) rename {src => skyvault/src}/vault/model/response/query/index.ts (70%) rename {src => skyvault/src}/vault/model/response/reidentify-text/index.ts (100%) rename {src => skyvault/src}/vault/model/response/tokenize/index.ts (65%) rename {src => skyvault/src}/vault/model/response/update/index.ts (64%) create mode 100644 skyvault/src/vault/skyflow/index.ts rename {src => skyvault/src}/vault/types/index.ts (72%) create mode 100644 skyvault/src/version.ts create mode 100644 skyvault/test/demo-credentials/empty.json create mode 100644 skyvault/test/demo-credentials/invalidJson.json create mode 100644 skyvault/test/demo-credentials/noClientId.json create mode 100644 skyvault/test/demo-credentials/noKeyId.json create mode 100644 skyvault/test/demo-credentials/noPrivateKey.json create mode 100644 skyvault/test/demo-credentials/noTokenURI.json create mode 100644 skyvault/test/demo-credentials/valid.json rename {test => skyvault/test}/mockData/pii.pdf (100%) rename {test => skyvault/test}/utils/validations.test.js (98%) rename {test => skyvault/test}/vault/client/client.test.js (99%) rename {test => skyvault/test}/vault/controller/connection.test.js (96%) rename {test => skyvault/test}/vault/controller/deprecated-detokenize.test.js (81%) rename {test => skyvault/test}/vault/controller/deprecated-insert.test.js (88%) rename {test => skyvault/test}/vault/controller/detect.test.js (97%) rename {test => skyvault/test}/vault/controller/vault.test.js (98%) rename {test => skyvault/test}/vault/model/deprecated.test.js (96%) rename {test => skyvault/test}/vault/skyflow/skyflow.test.js (100%) create mode 100644 skyvault/tsconfig.dts.json create mode 100644 skyvault/tsconfig.json create mode 100644 skyvault/tsup.config.ts delete mode 100644 src/error/messages/index.ts delete mode 100644 src/vault/config/vault/index.ts delete mode 100644 src/vault/skyflow/index.ts diff --git a/.github/workflows/beta-release.yml b/.github/workflows/beta-release.yml index a7f44e8f..6299155b 100644 --- a/.github/workflows/beta-release.yml +++ b/.github/workflows/beta-release.yml @@ -12,5 +12,7 @@ jobs: build-and-deploy: uses: ./.github/workflows/common-release.yml with: - tag: beta + product: skyflow-node + package-dir: skyvault + release-type: beta secrets: inherit diff --git a/.github/workflows/common-ci.yml b/.github/workflows/common-ci.yml index 9b79c71b..6eb5117d 100644 --- a/.github/workflows/common-ci.yml +++ b/.github/workflows/common-ci.yml @@ -22,6 +22,11 @@ jobs: - name: Run ESLint run: npm run eslint + # One command builds BOTH SDKs (common -> skyvault/flowvault via project references). + - name: Build all packages + run: npm run build + + # Fans out `npm test` across every workspace (common, skyvault, flowvault). - name: Run Tests env: SA_WITHOUT_CONTEXT: ${{ secrets.SA_WITHOUT_CONTEXT }} @@ -33,6 +38,6 @@ jobs: uses: codecov/codecov-action@v2.1.0 with: token: ${{ secrets.CODECOV_REPO_UPLOAD_TOKEN }} - files: coverage/lcov.info + files: skyvault/coverage/lcov.info,flowvault/coverage/lcov.info name: codecov-skyflow-node verbose: true diff --git a/.github/workflows/common-release.yml b/.github/workflows/common-release.yml index 2fad6eb9..c220c487 100644 --- a/.github/workflows/common-release.yml +++ b/.github/workflows/common-release.yml @@ -1,16 +1,30 @@ name: Common Release Workflow +# Reusable release pipeline, parameterized per product. +# product npm package name (skyflow-node | skyflow-flowvault-node) +# package-dir workspace dir (skyvault | flowvault) +# release-type public | beta | internal +# version semver to release (caller strips any tag prefix, e.g. flowvault/1.0.0 -> 1.0.0) + on: workflow_call: inputs: - tag: + product: + required: true + type: string + package-dir: + required: true + type: string + release-type: + required: true + type: string + version: required: false type: string jobs: build-sdk: runs-on: ubuntu-latest - steps: - uses: actions/checkout@v4 with: @@ -24,68 +38,68 @@ jobs: - name: Install Packages run: npm install --ignore-scripts - - name: Build - run: npm run build + - name: Resolve Version + id: version + run: | + if [ -n "${{ inputs.version }}" ]; then + echo "value=${{ inputs.version }}" >> $GITHUB_OUTPUT + else + # Fallback: strip an optional "/" prefix from the pushed tag. + echo "value=${GITHUB_REF_NAME##*/}" >> $GITHUB_OUTPUT + fi - - name: Get Previous Tag - id: previoustag - uses: WyriHaximus/github-action-get-previous-tag@v1 - with: - fallback: 1.0.0 + - name: Bump Version (${{ inputs.product }}) + run: | + chmod +x ./scripts/bump_version.sh + if [[ "${{ inputs.release-type }}" == "internal" ]]; then + ./scripts/bump_version.sh "${{ inputs.product }}" "${{ steps.version.outputs.value }}" "$(git rev-parse --short "$GITHUB_SHA")" + else + ./scripts/bump_version.sh "${{ inputs.product }}" "${{ steps.version.outputs.value }}" + fi + + # Build everything (common -> layers) then bundle the published package + # (inlines skyflow-common into a self-contained dist/). + - name: Build + run: | + npm run build + npm run build:dist --workspace ${{ inputs.package-dir }} - name: Resolve Branch Name id: resolve-branch - if: ${{ inputs.tag == 'beta' || inputs.tag == 'public' }} + if: ${{ inputs.release-type == 'beta' || inputs.release-type == 'public' }} run: | TAG_COMMIT=$(git rev-list -n 1 ${{ github.ref_name }}) - BRANCH_NAME=$(git branch -r --contains $TAG_COMMIT | grep -o 'origin/.*' | sed 's|origin/||' | head -n 1) - if [ -z "$BRANCH_NAME" ]; then echo "Error: Could not resolve branch for the tag." exit 1 fi - - echo "Resolved Branch Name: $BRANCH_NAME" echo "branch_name=$BRANCH_NAME" >> $GITHUB_ENV - - name: Bump Version - run: | - chmod +x ./scripts/bump_version.sh - if ${{ inputs.tag == 'internal' }}; then - ./scripts/bump_version.sh "${{ steps.previoustag.outputs.tag }}" "$(git rev-parse --short "$GITHUB_SHA")" - else - ./scripts/bump_version.sh "${{ steps.previoustag.outputs.tag }}" - fi - - name: Commit Changes run: | git config user.name ${{ github.actor }} git config user.email ${{ github.actor }}@users.noreply.github.com - if [[ "${{ inputs.tag }}" == "beta" || "${{ inputs.tag }}" == "public" ]]; then + if [[ "${{ inputs.release-type }}" == "beta" || "${{ inputs.release-type }}" == "public" ]]; then git checkout ${{ env.branch_name }} fi - git add package.json - if [[ "${{ inputs.tag }}" == "beta" || "${{ inputs.tag }}" == "public" ]]; then - git commit -m "[AUTOMATED] Release - ${{ steps.previoustag.outputs.tag }}" + git add ${{ inputs.package-dir }}/package.json ${{ inputs.package-dir }}/src/version.ts + if [[ "${{ inputs.release-type }}" == "beta" || "${{ inputs.release-type }}" == "public" ]]; then + git commit -m "[AUTOMATED] Release ${{ inputs.product }} - ${{ steps.version.outputs.value }}" || echo "nothing to commit" git push origin ${{ env.branch_name }} - elif [[ "${{ inputs.tag }}" == "internal" ]]; then - git commit -m "[AUTOMATED] Private Release ${{ steps.previoustag.outputs.tag }}-dev.$(git rev-parse --short $GITHUB_SHA)" + elif [[ "${{ inputs.release-type }}" == "internal" ]]; then + git commit -m "[AUTOMATED] Private Release ${{ inputs.product }} ${{ steps.version.outputs.value }}-dev.$(git rev-parse --short $GITHUB_SHA)" || echo "nothing to commit" git push origin ${{ github.ref_name }} -f fi - name: Publish + working-directory: ${{ inputs.package-dir }} run: | - if [[ "${{ inputs.tag }}" == "beta" ]]; then + if [[ "${{ inputs.release-type }}" == "beta" ]]; then npm publish --tag beta - elif [[ "${{ inputs.tag }}" == "public" ]]; then - MAJOR_VERSION=$(node -p "require('./package.json').version.split('.')[0]") - if [[ "$MAJOR_VERSION" == "1" ]]; then - npm publish --tag v1 - else - npm publish - fi - elif [[ "${{ inputs.tag }}" == "internal" ]]; then + elif [[ "${{ inputs.release-type }}" == "public" ]]; then + npm publish + elif [[ "${{ inputs.release-type }}" == "internal" ]]; then JFROG_AUTH=$(echo -n "$JFROG_USERNAME:$JFROG_PASSWORD" | base64 -w 0) npm config set registry https://prekarilabs.jfrog.io/prekarilabs/api/npm/npm/ npm config set //prekarilabs.jfrog.io/prekarilabs/api/npm/npm/:_auth "$JFROG_AUTH" diff --git a/.github/workflows/contract-tests.yml b/.github/workflows/contract-tests.yml index 58471087..db487535 100644 --- a/.github/workflows/contract-tests.yml +++ b/.github/workflows/contract-tests.yml @@ -24,7 +24,9 @@ jobs: - name: Install dependencies run: npm ci --ignore-scripts - - name: Verify API surface snapshot + # Builds every package, bundles the two published dist/ surfaces, then + # verifies BOTH SDKs' public-API snapshots (skyflow-node + skyflow-flowvault-node). + - name: Verify API surface snapshots (both SDKs) run: npm run contract-snapshot-verify - name: Show API surface diff @@ -34,12 +36,18 @@ jobs: echo "Lines prefixed with '-' were REMOVED from the public API." echo "Lines prefixed with '+' were ADDED to the public API." echo "" - diff -u api-report/skyflow-node.api.md temp/skyflow-node.api.md || true + echo "===== skyflow-node (skyvault) =====" + diff -u skyvault/api-report/skyflow-node.api.md skyvault/temp/skyflow-node.api.md || true + echo "" + echo "===== skyflow-flowvault-node (flowvault) =====" + diff -u flowvault/api-report/skyflow-flowvault-node.api.md flowvault/temp/skyflow-flowvault-node.api.md || true - - name: Upload API surface diff on failure + - name: Upload API surface diffs on failure if: failure() uses: actions/upload-artifact@v4 with: - name: api-surface-diff - path: temp/skyflow-node.api.md + name: api-surface-diffs + path: | + skyvault/temp/skyflow-node.api.md + flowvault/temp/skyflow-flowvault-node.api.md retention-days: 7 diff --git a/.github/workflows/flowvault-beta-release.yml b/.github/workflows/flowvault-beta-release.yml new file mode 100644 index 00000000..8bab6bbd --- /dev/null +++ b/.github/workflows/flowvault-beta-release.yml @@ -0,0 +1,18 @@ +name: FlowVault Public Beta Release + +on: + push: + tags: 'flowvault/*.*.*-beta.*' + paths-ignore: + - "package.json" + - "package-lock.json" + - "*.md" + +jobs: + build-and-deploy: + uses: ./.github/workflows/common-release.yml + with: + product: skyflow-flowvault-node + package-dir: flowvault + release-type: beta + secrets: inherit diff --git a/.github/workflows/flowvault-release.yml b/.github/workflows/flowvault-release.yml new file mode 100644 index 00000000..8f2423f2 --- /dev/null +++ b/.github/workflows/flowvault-release.yml @@ -0,0 +1,19 @@ +name: FlowVault Public Release + +# Prefixed tags select the FlowVault product: flowvault/x.y.z -> skyflow-flowvault-node. +on: + push: + tags: 'flowvault/[0-9]+.[0-9]+.[0-9]+' + paths-ignore: + - "package.json" + - "package-lock.json" + - "*.md" + +jobs: + call-common-workflow: + uses: ./.github/workflows/common-release.yml + with: + product: skyflow-flowvault-node + package-dir: flowvault + release-type: public + secrets: inherit diff --git a/.github/workflows/internal-release.yml b/.github/workflows/internal-release.yml index a87afa1d..8a21e2e8 100644 --- a/.github/workflows/internal-release.yml +++ b/.github/workflows/internal-release.yml @@ -1,5 +1,8 @@ name: Internal Release +# Branch-namespaced internal (JFrog) builds, routed per product: +# flowvault-release/* -> skyflow-flowvault-node +# skyvault-release/* | release/* -> skyflow-node (legacy release/* kept for skyvault) on: push: tags-ignore: @@ -10,10 +13,14 @@ on: - '*.md' branches: - release/* + - skyvault-release/* + - flowvault-release/* jobs: call-common-workflow: uses: ./.github/workflows/common-release.yml with: - tag: internal + product: ${{ startsWith(github.ref_name, 'flowvault-release/') && 'skyflow-flowvault-node' || 'skyflow-node' }} + package-dir: ${{ startsWith(github.ref_name, 'flowvault-release/') && 'flowvault' || 'skyvault' }} + release-type: internal secrets: inherit diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3f483a74..2b00c620 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -12,5 +12,7 @@ jobs: call-common-workflow: uses: ./.github/workflows/common-release.yml with: - tag: public + product: skyflow-node + package-dir: skyvault + release-type: public secrets: inherit diff --git a/.gitignore b/.gitignore index 2d76dc20..68f04726 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,7 @@ node_modules lib +dist +*.tsbuildinfo .vscode -coverage \ No newline at end of file +coverage +temp diff --git a/Rule/gitleaks.toml b/Rule/gitleaks.toml index e127f827..be258799 100644 --- a/Rule/gitleaks.toml +++ b/Rule/gitleaks.toml @@ -55,6 +55,13 @@ paths = [ '''\.gem$''', '''verification-metadata\.xml''', '''Database.refactorlog''', + # Maintainer-owned generated (Fern) clients contain example JWTs/tokens in + # their doc comments. Allowlist every generated dir across the workspaces + # (both the plain and the legacy leading-space directory names). + '''(?:^|/)_generated_(?:/|$)''', + '''(?:^|/) _generated_(?:/|$)''', + # Bundled publish output inlines the generated clients (example JWTs). + '''(?:^|/)dist/''', ] stopwords = [ "abcdefghijklmnopqrstuvwxyz", diff --git a/api-extractor.json b/api-extractor.json deleted file mode 100644 index 80bfdd6d..00000000 --- a/api-extractor.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "$schema": "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json", - "projectFolder": ".", - "mainEntryPointFilePath": "/lib/index.d.ts", - "bundledPackages": [], - "compiler": { - "tsconfigFilePath": "/tsconfig.json" - }, - "apiReport": { - "enabled": true, - "reportFolder": "/api-report/", - "reportTempFolder": "/temp/", - "reportFileName": ".api.md" - }, - "docModel": { - "enabled": false - }, - "dtsRollup": { - "enabled": false - }, - "tsdocMetadata": { - "enabled": false - }, - "messages": { - "compilerMessageReporting": { - "default": { - "logLevel": "warning" - } - }, - "extractorMessageReporting": { - "default": { - "logLevel": "warning" - }, - "ae-missing-release-tag": { - "logLevel": "none" - }, - "ae-setter-with-docs": { - "logLevel": "none" - } - }, - "tsdocMessageReporting": { - "default": { - "logLevel": "none" - } - } - } -} diff --git a/common/.babelrc b/common/.babelrc new file mode 100644 index 00000000..eb19681f --- /dev/null +++ b/common/.babelrc @@ -0,0 +1,14 @@ +{ + "presets": [ + ["@babel/preset-env", { + "targets": { + "browsers": ["last 2 versions", "ie >= 10"] + } + }] + ], + "sourceType": "unambiguous", + "plugins": [ + "@babel/plugin-transform-object-assign", + "@babel/plugin-transform-runtime" + ] +} \ No newline at end of file diff --git a/common/babel.config.js b/common/babel.config.js new file mode 100644 index 00000000..7bcee91b --- /dev/null +++ b/common/babel.config.js @@ -0,0 +1,32 @@ +/* + Copyright (c) 2022 Skyflow, Inc. +*/ +module.exports = function (api) { + api.cache(false); + const presets = [ + ['@babel/preset-typescript'], + [ + '@babel/preset-env', + { + corejs: { version: 3 }, + useBuiltIns: 'usage', + targets: { + edge: '17', + firefox: '60', + chrome: '67', + safari: '11.1', + ie: '11', + }, + }, + ], + ]; + const plugins = [ + ['@babel/plugin-proposal-decorators', { decoratorsBeforeExport: true }], + ['@babel/plugin-transform-class-properties'], + ['@babel/transform-runtime'], + ]; + return { + presets, + plugins, + }; +}; diff --git a/common/jest.config.js b/common/jest.config.js new file mode 100644 index 00000000..35014f57 --- /dev/null +++ b/common/jest.config.js @@ -0,0 +1,12 @@ +module.exports = { + transform: { + '^.+\\.(ts|tsx)?$': 'ts-jest', + '^.+\\.(js|jsx)$': 'babel-jest', + }, + verbose: true, + collectCoverage: false, + testEnvironmentOptions: { + "url": "https://skyflow-test.com" + }, + testEnvironment: "node", + }; diff --git a/common/package.json b/common/package.json new file mode 100644 index 00000000..e2fda3e8 --- /dev/null +++ b/common/package.json @@ -0,0 +1,23 @@ +{ + "name": "skyflow-common", + "version": "0.0.0", + "private": true, + "description": "Shared, contract-agnostic core for the Skyflow Node SDKs (internal; bundled into each published package).", + "main": "./lib/index.js", + "types": "./lib/index.d.ts", + "license": "MIT", + "scripts": { + "build": "tsc -b", + "test": "jest" + }, + "dependencies": { + "dotenv": "16.6.1", + "js-base64": "3.7.7", + "jsonwebtoken": "9.0.3", + "jwt-decode": "4.0.0", + "node-fetch": "2.7.0", + "qs": "6.15.2", + "readable-stream": "4.7.0", + "url-join": "4.0.1" + } +} diff --git a/common/src/_generated_/rest/Client.ts b/common/src/_generated_/rest/Client.ts new file mode 100644 index 00000000..b278e5b7 --- /dev/null +++ b/common/src/_generated_/rest/Client.ts @@ -0,0 +1,119 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "./environments"; +import * as core from "./core"; +import * as Skyflow from "./api/index"; +import urlJoin from "url-join"; +import * as errors from "./errors/index"; +import { Authentication } from "./api/resources/authentication/client/Client"; + +export declare namespace SkyflowClient { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class SkyflowClient { + protected _authentication: Authentication | undefined; + + constructor(protected readonly _options: SkyflowClient.Options) {} + + public get authentication(): Authentication { + return (this._authentication ??= new Authentication(this._options)); + } + + /** + * @param {string | undefined} vaultId + * @param {Skyflow.PatchV2VaultsIdRequest} request + * @param {SkyflowClient.RequestOptions} requestOptions - Request-specific configuration. + * + * @example + * await client.patchV2VaultsId(undefined) + */ + public patchV2VaultsId( + vaultId: string | undefined, + request: Skyflow.PatchV2VaultsIdRequest = {}, + requestOptions?: SkyflowClient.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__patchV2VaultsId(vaultId, request, requestOptions)); + } + + private async __patchV2VaultsId( + vaultId: string | undefined, + request: Skyflow.PatchV2VaultsIdRequest = {}, + requestOptions?: SkyflowClient.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + `v2/vaults/${encodeURIComponent(vaultId)}`, + ), + method: "PATCH", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: undefined, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling PATCH /v2/vaults/{vaultID}."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + return `Bearer ${await core.Supplier.get(this._options.token)}`; + } +} diff --git a/src/ _generated_/rest/api/resources/audit/client/index.ts b/common/src/_generated_/rest/api/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/client/index.ts rename to common/src/_generated_/rest/api/client/index.ts diff --git a/common/src/_generated_/rest/api/client/requests/PatchV2VaultsIdRequest.ts b/common/src/_generated_/rest/api/client/requests/PatchV2VaultsIdRequest.ts new file mode 100644 index 00000000..a2929a18 --- /dev/null +++ b/common/src/_generated_/rest/api/client/requests/PatchV2VaultsIdRequest.ts @@ -0,0 +1,9 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * {} + */ +export interface PatchV2VaultsIdRequest {} diff --git a/common/src/_generated_/rest/api/client/requests/index.ts b/common/src/_generated_/rest/api/client/requests/index.ts new file mode 100644 index 00000000..1a0e15b2 --- /dev/null +++ b/common/src/_generated_/rest/api/client/requests/index.ts @@ -0,0 +1 @@ +export { type PatchV2VaultsIdRequest } from "./PatchV2VaultsIdRequest"; diff --git a/common/src/_generated_/rest/api/errors/BadRequestError.ts b/common/src/_generated_/rest/api/errors/BadRequestError.ts new file mode 100644 index 00000000..e19131a1 --- /dev/null +++ b/common/src/_generated_/rest/api/errors/BadRequestError.ts @@ -0,0 +1,18 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as core from "../../core"; + +export class BadRequestError extends errors.SkyflowError { + constructor(body: Record, rawResponse?: core.RawResponse) { + super({ + message: "BadRequestError", + statusCode: 400, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, BadRequestError.prototype); + } +} diff --git a/common/src/_generated_/rest/api/errors/NotFoundError.ts b/common/src/_generated_/rest/api/errors/NotFoundError.ts new file mode 100644 index 00000000..ff59e84a --- /dev/null +++ b/common/src/_generated_/rest/api/errors/NotFoundError.ts @@ -0,0 +1,18 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as core from "../../core"; + +export class NotFoundError extends errors.SkyflowError { + constructor(body: Record, rawResponse?: core.RawResponse) { + super({ + message: "NotFoundError", + statusCode: 404, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, NotFoundError.prototype); + } +} diff --git a/common/src/_generated_/rest/api/errors/UnauthorizedError.ts b/common/src/_generated_/rest/api/errors/UnauthorizedError.ts new file mode 100644 index 00000000..73c2b114 --- /dev/null +++ b/common/src/_generated_/rest/api/errors/UnauthorizedError.ts @@ -0,0 +1,18 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as core from "../../core"; + +export class UnauthorizedError extends errors.SkyflowError { + constructor(body: Record, rawResponse?: core.RawResponse) { + super({ + message: "UnauthorizedError", + statusCode: 401, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, UnauthorizedError.prototype); + } +} diff --git a/common/src/_generated_/rest/api/errors/index.ts b/common/src/_generated_/rest/api/errors/index.ts new file mode 100644 index 00000000..91d5e383 --- /dev/null +++ b/common/src/_generated_/rest/api/errors/index.ts @@ -0,0 +1,3 @@ +export * from "./BadRequestError"; +export * from "./UnauthorizedError"; +export * from "./NotFoundError"; diff --git a/common/src/_generated_/rest/api/index.ts b/common/src/_generated_/rest/api/index.ts new file mode 100644 index 00000000..dba65abb --- /dev/null +++ b/common/src/_generated_/rest/api/index.ts @@ -0,0 +1,4 @@ +export * from "./types"; +export * from "./errors"; +export * from "./resources"; +export * from "./client"; diff --git a/common/src/_generated_/rest/api/resources/authentication/client/Client.ts b/common/src/_generated_/rest/api/resources/authentication/client/Client.ts new file mode 100644 index 00000000..7c3b0039 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/authentication/client/Client.ts @@ -0,0 +1,137 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace Authentication { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class Authentication { + constructor(protected readonly _options: Authentication.Options) {} + + /** + *

Generates a Bearer Token to authenticate with Skyflow. This method doesn't require the Authorization header.

Note: For recommended ways to authenticate, see API authentication.

+ * + * @param {Skyflow.V1GetAuthTokenRequest} request + * @param {Authentication.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.NotFoundError} + * + * @example + * await client.authentication.authenticationServiceGetAuthToken({ + * grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer", + * assertion: "eyLhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaXNzIjoiY29tcGFueSIsImV4cCI6MTYxNTE5MzgwNywiaWF0IjoxNjE1MTY1MDQwLCJhdWQiOiKzb21lYXVkaWVuY2UifQ.4pcPyMDQ9o1PSyXnrXCjTwXyr4BSezdI1AVTmud2fU3" + * }) + */ + public authenticationServiceGetAuthToken( + request: Skyflow.V1GetAuthTokenRequest, + requestOptions?: Authentication.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__authenticationServiceGetAuthToken(request, requestOptions)); + } + + private async __authenticationServiceGetAuthToken( + request: Skyflow.V1GetAuthTokenRequest, + requestOptions?: Authentication.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v1/auth/sa/oauth/token", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.V1GetAuthTokenResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Record, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Record, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Record, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v1/auth/sa/oauth/token."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + return `Bearer ${await core.Supplier.get(this._options.token)}`; + } +} diff --git a/src/ _generated_/rest/api/resources/authentication/client/index.ts b/common/src/_generated_/rest/api/resources/authentication/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/authentication/client/index.ts rename to common/src/_generated_/rest/api/resources/authentication/client/index.ts diff --git a/common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts b/common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts new file mode 100644 index 00000000..a1ef4fba --- /dev/null +++ b/common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts @@ -0,0 +1,25 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * { + * grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer", + * assertion: "eyLhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaXNzIjoiY29tcGFueSIsImV4cCI6MTYxNTE5MzgwNywiaWF0IjoxNjE1MTY1MDQwLCJhdWQiOiKzb21lYXVkaWVuY2UifQ.4pcPyMDQ9o1PSyXnrXCjTwXyr4BSezdI1AVTmud2fU3" + * } + */ +export interface V1GetAuthTokenRequest { + /** Grant type of the request. Set this to `urn:ietf:params:oauth:grant-type:jwt-bearer`. */ + grant_type: string; + /** User-signed JWT token that contains the following fields:
  • iss: Issuer of the JWT.
  • key: Unique identifier for the key.
  • aud: Recipient the JWT is intended for.
  • exp: Time the JWT expires.
  • sub: Subject of the JWT.
  • ctx: (Optional) Value for Context-aware authorization.
*/ + assertion: string; + /** Subject token. */ + subject_token?: string; + /** Subject token type. */ + subject_token_type?: string; + /** Token use type. Either `delegation` or `impersonation`. */ + requested_token_use?: string; + /** Subset of available roles to associate with the requested token. Uses the format "role:\ role:\". */ + scope?: string; +} diff --git a/src/ _generated_/rest/api/resources/authentication/client/requests/index.ts b/common/src/_generated_/rest/api/resources/authentication/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/authentication/client/requests/index.ts rename to common/src/_generated_/rest/api/resources/authentication/client/requests/index.ts diff --git a/src/ _generated_/rest/api/resources/authentication/index.ts b/common/src/_generated_/rest/api/resources/authentication/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/authentication/index.ts rename to common/src/_generated_/rest/api/resources/authentication/index.ts diff --git a/common/src/_generated_/rest/api/resources/index.ts b/common/src/_generated_/rest/api/resources/index.ts new file mode 100644 index 00000000..d933c827 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/index.ts @@ -0,0 +1,2 @@ +export * as authentication from "./authentication"; +export * from "./authentication/client/requests"; diff --git a/src/ _generated_/rest/api/types/GooglerpcStatus.ts b/common/src/_generated_/rest/api/types/GooglerpcStatus.ts similarity index 100% rename from src/ _generated_/rest/api/types/GooglerpcStatus.ts rename to common/src/_generated_/rest/api/types/GooglerpcStatus.ts diff --git a/src/ _generated_/rest/api/types/ProtobufAny.ts b/common/src/_generated_/rest/api/types/ProtobufAny.ts similarity index 100% rename from src/ _generated_/rest/api/types/ProtobufAny.ts rename to common/src/_generated_/rest/api/types/ProtobufAny.ts diff --git a/src/ _generated_/rest/api/types/V1GetAuthTokenResponse.ts b/common/src/_generated_/rest/api/types/V1GetAuthTokenResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1GetAuthTokenResponse.ts rename to common/src/_generated_/rest/api/types/V1GetAuthTokenResponse.ts diff --git a/common/src/_generated_/rest/api/types/index.ts b/common/src/_generated_/rest/api/types/index.ts new file mode 100644 index 00000000..63cf5e7a --- /dev/null +++ b/common/src/_generated_/rest/api/types/index.ts @@ -0,0 +1,3 @@ +export * from "./V1GetAuthTokenResponse"; +export * from "./GooglerpcStatus"; +export * from "./ProtobufAny"; diff --git a/src/ _generated_/rest/core/auth/BasicAuth.ts b/common/src/_generated_/rest/core/auth/BasicAuth.ts similarity index 100% rename from src/ _generated_/rest/core/auth/BasicAuth.ts rename to common/src/_generated_/rest/core/auth/BasicAuth.ts diff --git a/src/ _generated_/rest/core/auth/BearerToken.ts b/common/src/_generated_/rest/core/auth/BearerToken.ts similarity index 100% rename from src/ _generated_/rest/core/auth/BearerToken.ts rename to common/src/_generated_/rest/core/auth/BearerToken.ts diff --git a/src/ _generated_/rest/core/auth/index.ts b/common/src/_generated_/rest/core/auth/index.ts similarity index 100% rename from src/ _generated_/rest/core/auth/index.ts rename to common/src/_generated_/rest/core/auth/index.ts diff --git a/src/ _generated_/rest/core/fetcher/APIResponse.ts b/common/src/_generated_/rest/core/fetcher/APIResponse.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/APIResponse.ts rename to common/src/_generated_/rest/core/fetcher/APIResponse.ts diff --git a/src/ _generated_/rest/core/fetcher/Fetcher.ts b/common/src/_generated_/rest/core/fetcher/Fetcher.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/Fetcher.ts rename to common/src/_generated_/rest/core/fetcher/Fetcher.ts diff --git a/src/ _generated_/rest/core/fetcher/HttpResponsePromise.ts b/common/src/_generated_/rest/core/fetcher/HttpResponsePromise.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/HttpResponsePromise.ts rename to common/src/_generated_/rest/core/fetcher/HttpResponsePromise.ts diff --git a/src/ _generated_/rest/core/fetcher/RawResponse.ts b/common/src/_generated_/rest/core/fetcher/RawResponse.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/RawResponse.ts rename to common/src/_generated_/rest/core/fetcher/RawResponse.ts diff --git a/src/ _generated_/rest/core/fetcher/Supplier.ts b/common/src/_generated_/rest/core/fetcher/Supplier.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/Supplier.ts rename to common/src/_generated_/rest/core/fetcher/Supplier.ts diff --git a/src/ _generated_/rest/core/fetcher/createRequestUrl.ts b/common/src/_generated_/rest/core/fetcher/createRequestUrl.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/createRequestUrl.ts rename to common/src/_generated_/rest/core/fetcher/createRequestUrl.ts diff --git a/src/ _generated_/rest/core/fetcher/getFetchFn.ts b/common/src/_generated_/rest/core/fetcher/getFetchFn.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/getFetchFn.ts rename to common/src/_generated_/rest/core/fetcher/getFetchFn.ts diff --git a/src/ _generated_/rest/core/fetcher/getHeader.ts b/common/src/_generated_/rest/core/fetcher/getHeader.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/getHeader.ts rename to common/src/_generated_/rest/core/fetcher/getHeader.ts diff --git a/src/ _generated_/rest/core/fetcher/getRequestBody.ts b/common/src/_generated_/rest/core/fetcher/getRequestBody.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/getRequestBody.ts rename to common/src/_generated_/rest/core/fetcher/getRequestBody.ts diff --git a/src/ _generated_/rest/core/fetcher/getResponseBody.ts b/common/src/_generated_/rest/core/fetcher/getResponseBody.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/getResponseBody.ts rename to common/src/_generated_/rest/core/fetcher/getResponseBody.ts diff --git a/src/ _generated_/rest/core/fetcher/index.ts b/common/src/_generated_/rest/core/fetcher/index.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/index.ts rename to common/src/_generated_/rest/core/fetcher/index.ts diff --git a/src/ _generated_/rest/core/fetcher/makeRequest.ts b/common/src/_generated_/rest/core/fetcher/makeRequest.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/makeRequest.ts rename to common/src/_generated_/rest/core/fetcher/makeRequest.ts diff --git a/src/ _generated_/rest/core/fetcher/requestWithRetries.ts b/common/src/_generated_/rest/core/fetcher/requestWithRetries.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/requestWithRetries.ts rename to common/src/_generated_/rest/core/fetcher/requestWithRetries.ts diff --git a/src/ _generated_/rest/core/fetcher/signals.ts b/common/src/_generated_/rest/core/fetcher/signals.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/signals.ts rename to common/src/_generated_/rest/core/fetcher/signals.ts diff --git a/common/src/_generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts b/common/src/_generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts new file mode 100644 index 00000000..7d83aef2 --- /dev/null +++ b/common/src/_generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts @@ -0,0 +1,259 @@ +import type { Writable } from "readable-stream"; + +import { EventCallback, StreamWrapper } from "./chooseStreamWrapper"; + +export class Node18UniversalStreamWrapper< + ReadFormat extends Uint8Array | Uint16Array | Uint32Array, +> implements StreamWrapper< + Node18UniversalStreamWrapper | Writable | WritableStream, + ReadFormat +> { + private readableStream: ReadableStream; + private reader: ReadableStreamDefaultReader; + private events: Record; + private paused: boolean; + private resumeCallback: ((value?: unknown) => void) | null; + private encoding: string | null; + + constructor(readableStream: ReadableStream) { + this.readableStream = readableStream; + this.reader = this.readableStream.getReader(); + this.events = { + data: [], + end: [], + error: [], + readable: [], + close: [], + pause: [], + resume: [], + }; + this.paused = false; + this.resumeCallback = null; + this.encoding = null; + } + + public on(event: string, callback: EventCallback): void { + this.events[event]?.push(callback); + } + + public off(event: string, callback: EventCallback): void { + this.events[event] = this.events[event]?.filter((cb) => cb !== callback); + } + + public pipe( + dest: Node18UniversalStreamWrapper | Writable | WritableStream, + ): Node18UniversalStreamWrapper | Writable | WritableStream { + this.on("data", async (chunk) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._write(chunk); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } else { + dest.write(chunk); + } + }); + + this.on("end", async () => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._end(); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.close(); + } else { + dest.end(); + } + }); + + this.on("error", async (error) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._error(error); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.abort(error); + } else { + dest.destroy(error); + } + }); + + this._startReading(); + + return dest; + } + + public pipeTo( + dest: Node18UniversalStreamWrapper | Writable | WritableStream, + ): Node18UniversalStreamWrapper | Writable | WritableStream { + return this.pipe(dest); + } + + public unpipe(dest: Node18UniversalStreamWrapper | Writable | WritableStream): void { + this.off("data", async (chunk) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._write(chunk); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } else { + dest.write(chunk); + } + }); + + this.off("end", async () => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._end(); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.close(); + } else { + dest.end(); + } + }); + + this.off("error", async (error) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._error(error); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.abort(error); + } else { + dest.destroy(error); + } + }); + } + + public destroy(error?: Error): void { + this.reader + .cancel(error) + .then(() => { + this._emit("close"); + }) + .catch((err) => { + this._emit("error", err); + }); + } + + public pause(): void { + this.paused = true; + this._emit("pause"); + } + + public resume(): void { + if (this.paused) { + this.paused = false; + this._emit("resume"); + if (this.resumeCallback) { + this.resumeCallback(); + this.resumeCallback = null; + } + } + } + + public get isPaused(): boolean { + return this.paused; + } + + public async read(): Promise { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + + if (done) { + return undefined; + } + return value; + } + + public setEncoding(encoding: string): void { + this.encoding = encoding; + } + + public async text(): Promise { + const chunks: ReadFormat[] = []; + + while (true) { + const { done, value } = await this.reader.read(); + if (done) { + break; + } + if (value) { + chunks.push(value); + } + } + + const decoder = new TextDecoder(this.encoding || "utf-8"); + return decoder.decode(await new Blob(chunks).arrayBuffer()); + } + + public async json(): Promise { + const text = await this.text(); + return JSON.parse(text); + } + + private _write(chunk: ReadFormat): void { + this._emit("data", chunk); + } + + private _end(): void { + this._emit("end"); + } + + private _error(error: any): void { + this._emit("error", error); + } + + private _emit(event: string, data?: any): void { + if (this.events[event]) { + for (const callback of this.events[event] || []) { + callback(data); + } + } + } + + private async _startReading(): Promise { + try { + this._emit("readable"); + while (true) { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + this._emit("end"); + this._emit("close"); + break; + } + if (value) { + this._emit("data", value); + } + } + } catch (error) { + this._emit("error", error); + } + } + + [Symbol.asyncIterator](): AsyncIterableIterator { + return { + next: async () => { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + return { done: true, value: undefined }; + } + return { done: false, value }; + }, + [Symbol.asyncIterator]() { + return this; + }, + }; + } +} diff --git a/src/ _generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts b/common/src/_generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts rename to common/src/_generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts diff --git a/common/src/_generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts b/common/src/_generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts new file mode 100644 index 00000000..e5e8b680 --- /dev/null +++ b/common/src/_generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts @@ -0,0 +1,244 @@ +import { StreamWrapper } from "./chooseStreamWrapper"; + +type EventCallback = (data?: any) => void; + +export class UndiciStreamWrapper implements StreamWrapper< + UndiciStreamWrapper | WritableStream, + ReadFormat +> { + private readableStream: ReadableStream; + private reader: ReadableStreamDefaultReader; + private events: Record; + private paused: boolean; + private resumeCallback: ((value?: unknown) => void) | null; + private encoding: string | null; + + constructor(readableStream: ReadableStream) { + this.readableStream = readableStream; + this.reader = this.readableStream.getReader(); + this.events = { + data: [], + end: [], + error: [], + readable: [], + close: [], + pause: [], + resume: [], + }; + this.paused = false; + this.resumeCallback = null; + this.encoding = null; + } + + public on(event: string, callback: EventCallback): void { + this.events[event]?.push(callback); + } + + public off(event: string, callback: EventCallback): void { + this.events[event] = this.events[event]?.filter((cb) => cb !== callback); + } + + public pipe( + dest: UndiciStreamWrapper | WritableStream, + ): UndiciStreamWrapper | WritableStream { + this.on("data", (chunk) => { + if (dest instanceof UndiciStreamWrapper) { + dest._write(chunk); + } else { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } + }); + + this.on("end", () => { + if (dest instanceof UndiciStreamWrapper) { + dest._end(); + } else { + const writer = dest.getWriter(); + writer.close(); + } + }); + + this.on("error", (error) => { + if (dest instanceof UndiciStreamWrapper) { + dest._error(error); + } else { + const writer = dest.getWriter(); + writer.abort(error); + } + }); + + this._startReading(); + + return dest; + } + + public pipeTo( + dest: UndiciStreamWrapper | WritableStream, + ): UndiciStreamWrapper | WritableStream { + return this.pipe(dest); + } + + public unpipe(dest: UndiciStreamWrapper | WritableStream): void { + this.off("data", (chunk) => { + if (dest instanceof UndiciStreamWrapper) { + dest._write(chunk); + } else { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } + }); + + this.off("end", () => { + if (dest instanceof UndiciStreamWrapper) { + dest._end(); + } else { + const writer = dest.getWriter(); + writer.close(); + } + }); + + this.off("error", (error) => { + if (dest instanceof UndiciStreamWrapper) { + dest._error(error); + } else { + const writer = dest.getWriter(); + writer.abort(error); + } + }); + } + + public destroy(error?: Error): void { + this.reader + .cancel(error) + .then(() => { + this._emit("close"); + }) + .catch((err) => { + this._emit("error", err); + }); + } + + public pause(): void { + this.paused = true; + this._emit("pause"); + } + + public resume(): void { + if (this.paused) { + this.paused = false; + this._emit("resume"); + if (this.resumeCallback) { + this.resumeCallback(); + this.resumeCallback = null; + } + } + } + + public get isPaused(): boolean { + return this.paused; + } + + public async read(): Promise { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + return undefined; + } + return value; + } + + public setEncoding(encoding: string): void { + this.encoding = encoding; + } + + public async text(): Promise { + const chunks: BlobPart[] = []; + + while (true) { + const { done, value } = await this.reader.read(); + if (done) { + break; + } + if (value) { + chunks.push(value); + } + } + + const decoder = new TextDecoder(this.encoding || "utf-8"); + return decoder.decode(await new Blob(chunks).arrayBuffer()); + } + + public async json(): Promise { + const text = await this.text(); + return JSON.parse(text); + } + + private _write(chunk: ReadFormat): void { + this._emit("data", chunk); + } + + private _end(): void { + this._emit("end"); + } + + private _error(error: any): void { + this._emit("error", error); + } + + private _emit(event: string, data?: any): void { + if (this.events[event]) { + for (const callback of this.events[event] || []) { + callback(data); + } + } + } + + private async _startReading(): Promise { + try { + this._emit("readable"); + while (true) { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + this._emit("end"); + this._emit("close"); + break; + } + if (value) { + this._emit("data", value); + } + } + } catch (error) { + this._emit("error", error); + } + } + + [Symbol.asyncIterator](): AsyncIterableIterator { + return { + next: async () => { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + return { done: true, value: undefined }; + } + return { done: false, value }; + }, + [Symbol.asyncIterator]() { + return this; + }, + }; + } +} diff --git a/src/ _generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts b/common/src/_generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts rename to common/src/_generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts diff --git a/common/src/_generated_/rest/core/index.ts b/common/src/_generated_/rest/core/index.ts new file mode 100644 index 00000000..c752c6a6 --- /dev/null +++ b/common/src/_generated_/rest/core/index.ts @@ -0,0 +1,3 @@ +export * from "./fetcher"; +export * from "./auth"; +export * from "./runtime"; diff --git a/src/ _generated_/rest/core/json.ts b/common/src/_generated_/rest/core/json.ts similarity index 100% rename from src/ _generated_/rest/core/json.ts rename to common/src/_generated_/rest/core/json.ts diff --git a/src/ _generated_/rest/core/runtime/index.ts b/common/src/_generated_/rest/core/runtime/index.ts similarity index 100% rename from src/ _generated_/rest/core/runtime/index.ts rename to common/src/_generated_/rest/core/runtime/index.ts diff --git a/src/ _generated_/rest/core/runtime/runtime.ts b/common/src/_generated_/rest/core/runtime/runtime.ts similarity index 100% rename from src/ _generated_/rest/core/runtime/runtime.ts rename to common/src/_generated_/rest/core/runtime/runtime.ts diff --git a/common/src/_generated_/rest/environments.ts b/common/src/_generated_/rest/environments.ts new file mode 100644 index 00000000..4a1edc91 --- /dev/null +++ b/common/src/_generated_/rest/environments.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export const SkyflowEnvironment = { + Production: "https://manage.skyflowapis.com", + Sandbox: "https://manage.skyflowapis-preview.com", +} as const; + +export type SkyflowEnvironment = typeof SkyflowEnvironment.Production | typeof SkyflowEnvironment.Sandbox; diff --git a/src/ _generated_/rest/errors/SkyflowError.ts b/common/src/_generated_/rest/errors/SkyflowError.ts similarity index 100% rename from src/ _generated_/rest/errors/SkyflowError.ts rename to common/src/_generated_/rest/errors/SkyflowError.ts diff --git a/src/ _generated_/rest/errors/SkyflowTimeoutError.ts b/common/src/_generated_/rest/errors/SkyflowTimeoutError.ts similarity index 100% rename from src/ _generated_/rest/errors/SkyflowTimeoutError.ts rename to common/src/_generated_/rest/errors/SkyflowTimeoutError.ts diff --git a/src/ _generated_/rest/errors/index.ts b/common/src/_generated_/rest/errors/index.ts similarity index 100% rename from src/ _generated_/rest/errors/index.ts rename to common/src/_generated_/rest/errors/index.ts diff --git a/src/ _generated_/rest/index.ts b/common/src/_generated_/rest/index.ts similarity index 100% rename from src/ _generated_/rest/index.ts rename to common/src/_generated_/rest/index.ts diff --git a/src/ _generated_/rest/version.ts b/common/src/_generated_/rest/version.ts similarity index 100% rename from src/ _generated_/rest/version.ts rename to common/src/_generated_/rest/version.ts diff --git a/common/src/base/options.ts b/common/src/base/options.ts new file mode 100644 index 00000000..dc7e1ba5 --- /dev/null +++ b/common/src/base/options.ts @@ -0,0 +1,11 @@ +export class BaseOptions { + private _requestHeaders?: Record; + + setRequestHeaders(h: Record): void { + this._requestHeaders = h; + } + + getRequestHeaders(): Record | undefined { + return this._requestHeaders; + } +} diff --git a/common/src/base/request.ts b/common/src/base/request.ts new file mode 100644 index 00000000..1af86d47 --- /dev/null +++ b/common/src/base/request.ts @@ -0,0 +1,14 @@ +// Contract-agnostic marker base classes for request models. +// Layers (skyvault/flowvault) extend these for their own contract-specific requests. +export class BaseInsertRequest {} +export class BaseGetRequest {} +export class BaseDeleteRequest {} +export class BaseQueryRequest {} +export class BaseDetokenizeRequest {} +export class BaseUpdateRequest {} +export class BaseTokenizeRequest {} +export class BaseFileUploadRequest {} +export class BaseInvokeConnectionRequest {} +export class BaseGetTokensRequest {} +export class BaseUploadFilesRequest {} +export class BaseDeleteFilesRequest {} diff --git a/common/src/base/response.ts b/common/src/base/response.ts new file mode 100644 index 00000000..9fcc24b4 --- /dev/null +++ b/common/src/base/response.ts @@ -0,0 +1,14 @@ +// Contract-agnostic marker base classes for response models. +// Layers (skyvault/flowvault) extend these for their own contract-specific responses. +export class BaseInsertResponse {} +export class BaseGetResponse {} +export class BaseDeleteResponse {} +export class BaseQueryResponse {} +export class BaseDetokenizeResponse {} +export class BaseUpdateResponse {} +export class BaseTokenizeResponse {} +export class BaseFileUploadResponse {} +export class BaseInvokeConnectionResponse {} +export class BaseGetTokensResponse {} +export class BaseUploadFilesResponse {} +export class BaseDeleteFilesResponse {} diff --git a/src/vault/config/connection/index.ts b/common/src/config/connection/index.ts similarity index 100% rename from src/vault/config/connection/index.ts rename to common/src/config/connection/index.ts diff --git a/src/vault/config/credentials/index.ts b/common/src/config/credentials/index.ts similarity index 100% rename from src/vault/config/credentials/index.ts rename to common/src/config/credentials/index.ts diff --git a/common/src/config/vault/index.ts b/common/src/config/vault/index.ts new file mode 100644 index 00000000..0ede6fd0 --- /dev/null +++ b/common/src/config/vault/index.ts @@ -0,0 +1,18 @@ +// imports +import { Env } from "../../utils"; +import Credentials from "../credentials"; + +interface VaultConfig { + vaultId: string; + clusterId: string; + env?: Env; + credentials?: Credentials; + /** Overall per-call timeout in seconds (includes retries). Default 60. */ + timeout?: number; + /** Retry attempts after the first failure (408/429/5xx). Default 0 (off). */ + maxRetries?: number; + /** Override the vault URL derived from clusterId/env. */ + vaultUrl?: string; +} + +export default VaultConfig; diff --git a/src/error/codes/index.ts b/common/src/error/codes/index.ts similarity index 95% rename from src/error/codes/index.ts rename to common/src/error/codes/index.ts index 672bd33a..141bafd9 100644 --- a/src/error/codes/index.ts +++ b/common/src/error/codes/index.ts @@ -1,6 +1,6 @@ import errorMessages from "../messages"; -const SKYFLOW_ERROR_CODE = { +const _SKYFLOW_ERROR_CODE = { CONFIG_MISSING: { httpCode: 400, message: errorMessages.CONFIG_MISSING }, INVALID_TYPE_FOR_CONFIG: { httpCode: 400, message: errorMessages.INVALID_TYPE_FOR_CONFIG }, EMPTY_VAULT_CONFIG: { httpCode: 400, message: errorMessages.EMPTY_VAULT_CONFIG }, @@ -236,4 +236,24 @@ const SKYFLOW_ERROR_CODE = { INVALID_XML_FORMAT: { httpCode: 400, message: errorMessages.INVALID_XML_FORMAT }, }; +// Each access returns an entry whose `message` is resolved lazily from the +// identity-aware `errorMessages`, so the SDK version/name in error text reflects +// the live SDK identity (set per layer via setSdkIdentity) at throw time rather +// than being frozen at module-load. `httpCode` comes from the literal above. +const SKYFLOW_ERROR_CODE: typeof _SKYFLOW_ERROR_CODE = new Proxy(_SKYFLOW_ERROR_CODE, { + get(target, prop) { + if (typeof prop !== "string" || !(prop in target)) { + return (target as Record)[prop as string]; + } + const key = prop as keyof typeof _SKYFLOW_ERROR_CODE; + const httpCode = target[key].httpCode; + return { + httpCode, + get message(): string { + return (errorMessages as Record)[key as string]; + }, + }; + }, +}); + export default SKYFLOW_ERROR_CODE; \ No newline at end of file diff --git a/src/error/index.ts b/common/src/error/index.ts similarity index 100% rename from src/error/index.ts rename to common/src/error/index.ts diff --git a/common/src/error/messages/index.ts b/common/src/error/messages/index.ts new file mode 100644 index 00000000..b5c193ca --- /dev/null +++ b/common/src/error/messages/index.ts @@ -0,0 +1,252 @@ +import { getSdkIdentity } from "../../identity"; + +const errorPrefix = (): string => `Skyflow Node SDK v${getSdkIdentity().version}`; + +const errorMessages = { + get CONFIG_MISSING(): string { return `${errorPrefix()} Initialization failed. Skyflow config cannot be empty. Specify a valid skyflow config.`; }, + get INVALID_SKYFLOW_CONFIG(): string { return `${errorPrefix()} Initialization failed. Invalid skyflow config. Vault/Connection config key missing in skyflow config.`; }, + get INVALID_TYPE_FOR_CONFIG(): string { return `${errorPrefix()} Initialization failed. Invalid %s1 config. Specify a valid %s1 config.`; }, + get EMPTY_VAULT_CONFIG(): string { return `${errorPrefix()} Initialization failed. Vault config cannot be empty. Specify a valid vault config.`; }, + get EMPTY_CONNECTION_CONFIG(): string { return `${errorPrefix()} Initialization failed. Connection config cannot be empty. Specify a valid connection config.`; }, + + get EMPTY_VAULT_ID(): string { return `${errorPrefix()} Initialization failed. Invalid vault ID. Specify a valid vault Id.`; }, + get EMPTY_VAULT_ID_VALIDATION(): string { return `${errorPrefix()} Validation error. Invalid vault ID. Specify a valid vault Id.`; }, + get INVALID_VAULT_ID(): string { return `${errorPrefix()} Initialization failed. Invalid vault ID. Specify a valid vault Id as a string.`; }, + get EMPTY_CLUSTER_ID(): string { return `${errorPrefix()} Initialization failed. Invalid cluster ID. Specify a valid cluster Id for vault with vaultId %s1 .`; }, + get INVALID_CLUSTER_ID(): string { return `${errorPrefix()} Initialization failed. Invalid cluster ID. Specify cluster Id as a string for vault with vaultId %s1.`; }, + get INVALID_TOKEN(): string { return `${errorPrefix()} Validation error. Invalid token. Specify a valid token.`; }, + get TOKEN_EXPIRED(): string { return `${errorPrefix()} Validation error. Token provided is either invalid or has expired. Specify a valid token.`; }, + get INVALID_ENV(): string { return `${errorPrefix()} Initialization failed. Invalid env. Specify a valid env for vault with vaultId %s1.`; }, + get INVALID_LOG_LEVEL(): string { return `${errorPrefix()} Initialization failed. Invalid log level. Specify a valid log level.`; }, + get EMPTY_CREDENTIAL_FILE_PATH(): string { return `${errorPrefix()} Initialization failed. Invalid credentials. Specify a valid file path.`; }, + get INVALID_CREDENTIAL_FILE_PATH(): string { return `${errorPrefix()} Initialization failed. Invalid credentials. Expected file path to be a string.`; }, + + get INVALID_CREDENTIALS_FILE_PATH(): string { return `${errorPrefix()} Initialization failed. Invalid skyflow credentials. Expected file path to exists.`; }, + get INVALID_TOKEN_URI(): string { return `${errorPrefix()} Initialization failed. Invalid Skyflow credentials. The token URI must be a string and a valid URL.`; }, + get INVALID_TOKEN_URI_WITH_ID(): string { return `${errorPrefix()} Initialization failed. Invalid Skyflow credentials. The token URI must be a string and a valid URL for %s1 with %s2 %s3.`; }, + get INVALID_KEY(): string { return `${errorPrefix()} Initialization failed. Invalid skyflow credentials. Specify a valid api key.`; }, + get INVALID_PARSED_CREDENTIALS_STRING(): string { return `${errorPrefix()} Initialization failed. Invalid skyflow credentials. Specify a valid credentials string.`; }, + get INVALID_BEARER_TOKEN(): string { return `${errorPrefix()} Initialization failed. Invalid skyflow credentials. Bearer token is invalid or expired. Specify a valid token.`; }, + + get INVALID_FILE_PATH_WITH_ID(): string { return `${errorPrefix()} Initialization failed. Invalid credentials. Expected file path to exists for %s1 with %s2 %s3.`; }, + get INVALID_KEY_WITH_ID(): string { return `${errorPrefix()} Initialization failed. Invalid credentials. Specify a valid api key for %s1 with %s2 %s3.`; }, + get INVALID_PARSED_CREDENTIALS_STRING_WITH_ID(): string { return `${errorPrefix()} Initialization failed. Invalid credentials. Specify a valid credentials string for %s1 with %s2 %s3.`; }, + get INVALID_BEARER_TOKEN_WITH_ID(): string { return `${errorPrefix()} Initialization failed. Invalid credentials. Bearer token is invalid or expired. Specify a valid token for %s1 with %s2 %s3.`; }, + + get EMPTY_CONNECTION_ID_VALIDATION(): string { return `${errorPrefix()} Validation error. Invalid connection ID. Specify a valid connection Id.`; }, + get EMPTY_CONNECTION_ID(): string { return `${errorPrefix()} Initialization failed. Invalid connection ID. Specify a valid connection Id.`; }, + get INVALID_CONNECTION_ID(): string { return `${errorPrefix()} Initialization failed. Invalid connection ID. Specify connection Id as a string.`; }, + get EMPTY_CONNECTION_URL(): string { return `${errorPrefix()} Initialization failed. Invalid connection URL. Specify a valid connection Url.`; }, + get INVALID_CONNECTION_URL(): string { return `${errorPrefix()} Initialization failed. Invalid connection URL. Specify connection Url as a valid url.`; }, + + get VAULT_ID_EXITS_IN_CONFIG_LIST(): string { return `${errorPrefix()} Validation error. %s1 already exists in the config list. Specify a new vaultId.`; }, + get CONNECTION_ID_EXITS_IN_CONFIG_LIST(): string { return `${errorPrefix()} Validation error. %s1 already exists in the config list. Specify a new vaultId.`; }, + get VAULT_ID_NOT_IN_CONFIG_LIST(): string { return `${errorPrefix()} Validation error. %s1 is missing from the config. Specify the vaultId's from config.`; }, + get CONNECTION_ID_NOT_IN_CONFIG_LIST(): string { return `${errorPrefix()} Validation error. %s1 is missing from the config. Specify the connectionIds from config.`; }, + + get EMPTY_CREDENTIALS(): string { return `${errorPrefix()} Validation error. Invalid credentials. Credentials must not be empty.`; }, + get INVALID_CREDENTIALS(): string { return `${errorPrefix()} Validation error. Invalid credentials. Specify a valid credentials.`; }, + get CREDENTIALS_WITH_NO_VALID_KEY(): string { return `${errorPrefix()} Validation error. Invalid credentials. Credentials must include one of the following: { apiKey, token, credentials, path }.`; }, + get MULTIPLE_CREDENTIALS_PASSED(): string { return `${errorPrefix()} Validation error. Multiple credentials provided. Specify only one of the following: { apiKey, token, credentials, path }.`; }, + get INVALID_CREDENTIALS_WITH_ID(): string { return `${errorPrefix()} Validation error. Invalid credentials. Credentials must include one of the following: { apiKey, token, credentials, path } for %s1 with %s2 %s3.`; }, + get MULTIPLE_CREDENTIALS_PASSED_WITH_ID(): string { return `${errorPrefix()} Validation error. Invalid credentials.Specify only one of the following: { apiKey, token, credentials, path } for %s1 with %s2 %s3.`; }, + + get FILE_NOT_FOUND(): string { return `${errorPrefix()} Validation error. Credential file not found at %s1. Verify the file path.`; }, + get INVALID_JSON_FILE(): string { return `${errorPrefix()} Validation error. File at %s1 is not in valid JSON format. Verify the file contents.`; }, + + get EMPTY_CREDENTIALS_STRING(): string { return `${errorPrefix()} Validation error. Invalid credentials. Specify a valid credentials.`; }, + get INVALID_CREDENTIALS_STRING(): string { return `${errorPrefix()} Validation error. Invalid credentials. Specify credentials as a string.`; }, + + get MISSING_PRIVATE_KEY(): string { return `${errorPrefix()} Validation error. Unable to read private key in credentials. Verify your private key.`; }, + get MISSING_CLIENT_ID(): string { return `${errorPrefix()} Validation error. Unable to read client ID in credentials. Verify your client ID.`; }, + get MISSING_KEY_ID(): string { return `${errorPrefix()} Validation error. Unable to read key ID in credentials. Verify your key ID.`; }, + get MISSING_TOKEN_URI(): string { return `${errorPrefix()} Validation error. Unable to read token URI in credentials. Verify your token URI.`; }, + + get INVALID_ROLES_KEY_TYPE(): string { return `${errorPrefix()} Validation error. Invalid roles. Specify roles as an array.`; }, + get INVALID_CONTEXT(): string { return `${errorPrefix()} Validation error. Invalid context. Specify context as a string.`; }, + get EMPTY_ROLES(): string { return `${errorPrefix()} Validation error. Invalid roles. Specify at least one role.`; }, + + get INVALID_JSON_FORMAT(): string { return `${errorPrefix()} Validation error. Credentials is not in valid JSON format. Verify the credentials.`; }, + + get EMPTY_DATA_TOKENS(): string { return `${errorPrefix()} Validation error. Invalid data tokens. Specify valid data tokens.`; }, + get DATA_TOKEN_KEY_TYPE(): string { return `${errorPrefix()} Validation error. Invalid data tokens. Specify data token as an string array.`; }, + get TIME_TO_LIVE_KET_TYPE(): string { return `${errorPrefix()} Validation error. Invalid time to live. Specify time to live parameter as an string.`; }, + + get INVALID_DELETE_IDS_INPUT(): string { return `${errorPrefix()} Validation error. Invalid delete ids type in delete request. Specify delete ids as a string array.`; }, + get EMPTY_DELETE_IDS(): string { return `${errorPrefix()} Validation error. Delete ids array cannot be empty. Specify id's in delete request.`; }, + get INVALID_ID_IN_DELETE(): string { return `${errorPrefix()} Validation error. Invalid type of id passed in delete request. Id must be of type string at index %s1.`; }, + get INVALID_DELETE_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid delete request. Specify a valid delete request.`; }, + get EMPTY_ID_IN_DELETE(): string { return `${errorPrefix()} Validation error. Id cannot be empty in delete request. Specify a valid id.`; }, + + get MISSING_REDACTION_TYPE_IN_DETOKENIZE(): string { return `${errorPrefix()} Validation error. Redaction type cannot be empty in detokenize request. Specify the redaction type.`; }, + get INVALID_REDACTION_TYPE_IN_DETOKENIZE(): string { return `${errorPrefix()} Validation error. Invalid redaction type in detokenize request. Specify a redaction type.`; }, + get INVALID_TOKENS_TYPE_IN_DETOKENIZE(): string { return `${errorPrefix()} Validation error. Invalid tokens type in detokenize request. Specify tokens as a string array.`; }, + get EMPTY_TOKENS_IN_DETOKENIZE(): string { return `${errorPrefix()} Validation error. Tokens array cannot be empty. Specify token's in detokenize request.`; }, + get EMPTY_TOKEN_IN_DETOKENIZE(): string { return `${errorPrefix()} Validation error. Token cannot be empty in detokenize request. Specify a valid token at index %s1.`; }, + get INVALID_TOKEN_IN_DETOKENIZE(): string { return `${errorPrefix()} Validation error. Invalid type of token passed in detokenize request. token must be of type string at index %s1.`; }, + get INVALID_DETOKENIZE_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid detokenize request. Specify a valid detokenize request.`; }, + + get MISSING_VALUES_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Values cannot be empty in tokenize request. Specify valid values.`; }, + get INVALID_VALUES_TYPE_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Invalid values type in tokenize request. Specify valid values of type array.`; }, + get EMPTY_VALUES_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Values array cannot be empty. Specify value's in tokenize request.`; }, + get EMPTY_DATA_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Data cannot be empty in tokenize request. Specify a valid data at index %s1.`; }, + get INVALID_DATA_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Invalid Data. Specify a valid data at index %s1.`; }, + get INVALID_COLUMN_GROUP_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Invalid type of column group passed in tokenize request. Column group must be of type string at index %s1.`; }, + get INVALID_VALUE_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Invalid type of value passed in tokenize request. Value must be of type string at index %s1.`; }, + get INVALID_TOKENIZE_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid tokenize request. Specify a valid tokenize request.`; }, + get EMPTY_COLUMN_GROUP_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Column group cannot be empty in tokenize request. Specify a valid column group at index %s1.`; }, + get EMPTY_VALUE_IN_TOKENIZE(): string { return `${errorPrefix()} Validation error. Value cannot be empty in tokenize request. Specify a valid value at index %s1.`; }, + + get INVALID_RECORD_IN_INSERT(): string { return `${errorPrefix()} Validation error. Invalid data in insert request. data must be of type object at index %s1.`; }, + get INVALID_RECORD_IN_UPDATE(): string { return `${errorPrefix()} Validation error. Invalid data in update request. data must be of type object.`; }, + get EMPTY_RECORD_IN_INSERT(): string { return `${errorPrefix()} Validation error. Data cannot be empty in insert request. Specify valid data at index %s1.`; }, + get INVALID_INSERT_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid insert request. Specify a valid insert request.`; }, + get INVALID_TYPE_OF_DATA_IN_INSERT(): string { return `${errorPrefix()} Validation error. Invalid type of data in insert request. Specify data as a object array.`; }, + get EMPTY_DATA_IN_INSERT(): string { return `${errorPrefix()} Validation error. Data array cannot be empty. Specify data in insert request.`; }, + + get INVALID_QUERY_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid query request. Specify a valid query request.`; }, + get EMPTY_QUERY(): string { return `${errorPrefix()} Validation error. Query cannot be empty in query request. Specify a valid query.`; }, + get INVALID_QUERY(): string { return `${errorPrefix()} Validation error. Invalid query in query request. Specify a valid query.`; }, + + get INVALID_FILE_UPLOAD_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid file upload request. Specify a valid file upload request.`; }, + get MISSING_TABLE_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Table name cannot be empty in file upload request. Specify table name as a string.`; }, + get INVALID_TABLE_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Invalid table name in file upload request. Specify a valid table name.`; }, + get MISSING_SKYFLOW_ID_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Skyflow id cannot be empty in file upload request. Specify a valid skyflow Id as string.`; }, + get INVALID_SKYFLOW_ID_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Invalid skyflow Id in file upload request. Specify a valid skyflow Id.`; }, + get MISSING_COLUMN_NAME_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Column name cannot be empty in file upload request. Specify a valid column name as string.`; }, + get INVALID_COLUMN_NAME_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Invalid column name in file upload request. Specify a valid column name.`; }, + get MISSING_FILE_PATH_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. File path cannot be empty in file upload request. Specify a valid file path as string.`; }, + get INVALID_FILE_PATH_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Invalid file path in file upload request. Specify a valid file path.`; }, + get MISSING_FILE_SOURCE_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Provide exactly one of filePath, base64, or fileObject.`; }, + get MISSING_FILE_NAME_FOR_BASE64(): string { return `${errorPrefix()} Validation error. File name is required when providing a base64 string`; }, + get INVALID_FILE_OBJECT_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Invalid file object in file upload request. Specify a valid file object.`; }, + get MISSING_FILE_NAME_IN_FILE_OBJECT(): string { return `${errorPrefix()} Validation error. File name is required when providing a file object. Specify a valid file name.`; }, + get INVALID_BASE64_IN_UPLOAD_FILE(): string { return `${errorPrefix()} Validation error. Invalid base64 string in file upload request. Specify a valid base64 string.`; }, + + get MISSING_SKYFLOW_ID_IN_UPDATE(): string { return `${errorPrefix()} Validation error. Skyflow id name cannot be empty in update request. Specify a skyflow Id name as string.`; }, + get INVALID_SKYFLOW_ID_IN_UPDATE(): string { return `${errorPrefix()} Validation error. Invalid skyflow Id in update request. Specify a valid skyflow Id.`; }, + get INVALID_TYPE_OF_UPDATE_DATA(): string { return `${errorPrefix()} Validation error. Invalid update data in update request. Specify a valid update data as array of object.`; }, + get EMPTY_UPDATE_DATA(): string { return `${errorPrefix()} Validation error. Update data cannot be empty in update request. Specify a valid update data.`; }, + get INVALID_UPDATE_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid update request. Specify a valid update request.`; }, + get INVALID_DATA_IN_UPDATE(): string { return `${errorPrefix()} Validation error. Invalid data in update request. data must be of type object at index %s1.`; }, + get EMPTY_DATA_IN_UPDATE(): string { return `${errorPrefix()} Validation error. Data cannot be empty in update request. Specify a valid data at index %s1.`; }, + get INVALID_UPDATE_TOKENS(): string { return `${errorPrefix()} Validation error. Invalid tokens. Specify valid tokens as object.`; }, + get INVALID_TOKEN_IN_UPDATE(): string { return `${errorPrefix()} Validation error. Invalid tokens. Specify valid tokens as key value pairs.`; }, + + get EMPTY_TABLE_NAME(): string { return `${errorPrefix()} Validation error. Table name cannot be empty. Specify a valid table name.`; }, + get INVALID_TABLE_NAME(): string { return `${errorPrefix()} Validation error. Invalid table name. Specify a valid table name as string.`; }, + get EMPTY_REDACTION_TYPE(): string { return `${errorPrefix()} Validation error. Redaction type cannot be empty. Specify a valid redaction type.`; }, + get INVALID_REDACTION_TYPE(): string { return `${errorPrefix()} Validation error. Invalid redaction type. Specify a valid redaction type.`; }, + + get INVALID_TYPE_OF_IDS(): string { return `${errorPrefix()} Validation error. Invalid ids passed in get request. Specify valid ids as array of string.`; }, + get EMPTY_IDS_IN_GET(): string { return `${errorPrefix()} Validation error. Ids cannot be empty in get request. Specify valid ids.`; }, + get EMPTY_ID_IN_GET(): string { return `${errorPrefix()} Validation error. Id cannot be empty. Specify a valid Id at index %s1.`; }, + get INVALID_ID_IN_GET(): string { return `${errorPrefix()} Validation error. Invalid Id. Specify a valid Id at index %s1 as string.`; }, + get INVALID_GET_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid get request. Specify a valid get request.`; }, + + get EMPTY_COLUMN_NAME(): string { return `${errorPrefix()} Validation error. Column name cannot be empty. Specify a valid column name.`; }, + get INVALID_COLUMN_NAME(): string { return `${errorPrefix()} Validation error. Invalid column name. Specify a valid column name as string.`; }, + get INVALID_COLUMN_VALUES(): string { return `${errorPrefix()} Validation error. Invalid column values. Specify valid column values as string array.`; }, + get EMPTY_COLUMN_VALUES(): string { return `${errorPrefix()} Validation error. Column values cannot be empty. Specify valid column values.`; }, + get EMPTY_COLUMN_VALUE(): string { return `${errorPrefix()} Validation error. Column value cannot be empty. Specify a valid column value at index %s1.`; }, + get INVALID_COLUMN_VALUE(): string { return `${errorPrefix()} Validation error. Invalid column value. Specify a valid column value at index %s1 as string.`; }, + get INVALID_GET_COLUMN_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid get column request. Specify a valid get column request.`; }, + + get EMPTY_URL(): string { return `${errorPrefix()} Validation error. Url cannot be empty. Specify a valid url.`; }, + get INVALID_URL(): string { return `${errorPrefix()} Validation error. Invalid url. Specify a valid url as a string.`; }, + get EMPTY_METHOD_NAME(): string { return `${errorPrefix()} Validation error. Method name cannot be empty. Specify a valid method name.`; }, + get INVALID_METHOD_NAME(): string { return `${errorPrefix()} Validation error. Invalid method name. Specify a valid method name as a string.`; }, + + get EMPTY_PATH_PARAMS(): string { return `${errorPrefix()} Validation error. Path params cannot be empty. Specify valid path params.`; }, + get INVALID_PATH_PARAMS(): string { return `${errorPrefix()} Validation error. Invalid path params. Specify valid path params.`; }, + get EMPTY_QUERY_PARAMS(): string { return `${errorPrefix()} Validation error. Query params cannot be empty. Specify valid query params.`; }, + get INVALID_QUERY_PARAMS(): string { return `${errorPrefix()} Validation error. Invalid query params. Specify valid query params.`; }, + get EMPTY_BODY(): string { return `${errorPrefix()} Validation error. Body cannot be empty. Specify a valid body.`; }, + get INVALID_BODY(): string { return `${errorPrefix()} Validation error. Invalid body. Specify a valid body.`; }, + get EMPTY_HEADERS(): string { return `${errorPrefix()} Validation error. Headers cannot be empty. Specify valid headers.`; }, + get INVALID_HEADERS(): string { return `${errorPrefix()} Validation error. Invalid headers. Specify valid headers.`; }, + + get INVALID_INVOKE_CONNECTION_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid invoke connection request. Specify a valid get invoke connection request.`; }, + get ERROR_OCCURRED(): string { return `${errorPrefix()} API error. Error occurred.`; }, + + get EMPTY_INSERT_TOKEN(): string { return `${errorPrefix()} Validation error. Tokens object cannot be empty. Specify a valid tokens object at index %s1.`; }, + get INVALID_INSERT_TOKEN(): string { return `${errorPrefix()} Validation error. Invalid tokens object. Specify a valid tokens object at index %s1.`; }, + get INVALID_INSERT_TOKENS(): string { return `${errorPrefix()} Validation error. Invalid tokens. Specify valid tokens as object array.`; }, + get INVALID_TOKEN_MODE(): string { return `${errorPrefix()} Validation error. The token mode key has a value of type %s1. Specify token as type of TokenMode.`; }, + get INVALID_HOMOGENEOUS(): string { return `${errorPrefix()} Validation error. The homogeneous key has a value of type %s1. Specify homogeneous as boolean.`; }, + get INVALID_TOKEN_STRICT(): string { return `${errorPrefix()} Validation error. The tokenStrict key has a value of type %s1. Specify tokenStrict as boolean.`; }, + get INVALID_CONTINUE_ON_ERROR(): string { return `${errorPrefix()} Validation error. The continueOnError key has a value of type %s1. Specify continueOnError as boolean.`; }, + get INVALID_UPSERT(): string { return `${errorPrefix()} Validation error. The upsert key has a value of type %s1. Specify upsert as string.`; }, + get INVALID_RETURN_TOKEN(): string { return `${errorPrefix()} Validation error. The returnToken key has a value of type %s1. Specify returnToken as boolean.`; }, + + get NO_TOKENS_WITH_TOKEN_MODE(): string { return `${errorPrefix()} Validation error. Tokens weren't specified for records while 'tokenMode' was ENABLE or ENABLE_STRICT`; }, + get INSUFFICIENT_TOKENS_PASSED_FOR_TOKEN_MODE_ENABLE_STRICT(): string { return `${errorPrefix()} Validation error. 'tokenMode' is set to 'ENABLE_STRICT', but some fields are missing tokens. Specify tokens for all fields.`; }, + + get INVALID_DOWNLOAD_URL(): string { return `${errorPrefix()} Validation error. The downloadURL key has a value of type %s1. Specify downloadURL as string.`; }, + + get EMPTY_FIELD(): string { return `${errorPrefix()} Validation error. Filed value cannot be empty. Specify a valid filed value at index %s1.`; }, + get INVALID_FIELD(): string { return `${errorPrefix()} Validation error. Invalid filed value. Specify a valid filed value at index %s1 as string.`; }, + + get INVALID_OFFSET(): string { return `${errorPrefix()} Validation error. The offset key has a value of type %s1. Specify offset as string.`; }, + get INVALID_LIMIT(): string { return `${errorPrefix()} Validation error. The limit key has a value of type %s1. Specify limit as string.`; }, + + get INVALID_ORDER_BY(): string { return `${errorPrefix()} Validation error. The orderBy key has a value of type %s1. Specify orderBy as string.`; }, + get INVALID_FIELDS(): string { return `${errorPrefix()} Validation error. The fields key has a value of type %s1. Specify fields as array of strings.`; }, + + get INVAILD_JSON_RESPONSE(): string { return `${errorPrefix()} Validation error. The invalid json response. Please reach out to skyflow using requestId - %s1.`; }, + + get EMPTY_VAULT_CLIENTS(): string { return `${errorPrefix()} Validation error. No vault config found. Please add a vault config`; }, + get EMPTY_CONNECTION_CLIENTS(): string { return `${errorPrefix()} Validation error. No connection config found. Please add a connection config`; }, + + get GENERIC_API_ERROR(): string { return `${errorPrefix()} Validation error. Invalid configuration. Please add a valid vault configuration`; }, + get INVALID_TEXT_IN_DEIDENTIFY(): string { return `${errorPrefix()} Validation error. The text field is required and must be a non-empty string. Specify a valid text.`; }, + get INVALID_ENTITIES_IN_DEIDENTIFY(): string { return `${errorPrefix()} Validation error. The entities field must be an array of DetectEntities enums. Specify a valid entities.`; }, + get INVALID_ALLOW_REGEX_LIST(): string { return `${errorPrefix()} Validation error. The allowRegexList field must be an array of strings. Specify a valid allowRegexList.`; }, + get INVALID_RESTRICT_REGEX_LIST(): string { return `${errorPrefix()} Validation error. The restrictRegexList field must be an array of strings. Specify a valid restrictRegexList.`; }, + get INVALID_TOKEN_FORMAT(): string { return `${errorPrefix()} Validation error. The tokenFormat key must be an instance of TokenFormat. Specify a valid token format.`; }, + get INVALID_TRANSFORMATIONS(): string { return `${errorPrefix()} Validation error. The transformations key must be an instance of Transformations. Specify a valid transformations.`; }, + get VAULT_TOKEN_FORMAT_NOT_ALLOWED_FOR_DEIDENTIFY_FILES(): string { return `${errorPrefix()} Validation error. Vault token format is not allowed for deidentify file request.`; }, + + get INVALID_TEXT_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The text field is required and must be a non-empty string. Specify a valid text.`; }, + get INVALID_REDACTED_ENTITIES_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The redactedEntities field must be an array of DetectEntities enums. Specify a valid redactedEntities.`; }, + get INVALID_MASKED_ENTITIES_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The maskedEntities field must be an array of DetectEntities enums. Specify a valid maskedEntities.`; }, + get INVALID_PLAIN_TEXT_ENTITIES_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The plainTextEntities field must be an array of DetectEntities enums. Specify a valid plainTextEntities.`; }, + + get INVALID_DEIDENTIFY_FILE_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid deidentify file request. Specify a valid deidentify file request.`; }, + get INVALID_DEIDENTIFY_FILE_INPUT(): string { return `${errorPrefix()} Validation error. Invalid deidentify file input. Please provide either a file or a file path.`; }, + get EMPTY_FILE_OBJECT(): string { return `${errorPrefix()} Validation error. File object cannot be empty. Specify a valid file object.`; }, + get INVALID_FILE_FORMAT(): string { return `${errorPrefix()} Validation error. Invalid file format. Specify a valid file format.`; }, + get MISSING_FILE_SOURCE(): string { return `${errorPrefix()} Validation error. Provide exactly one of filePath, base64, or fileObject.`; }, + get INVALID_FILE_OBJECT(): string { return `${errorPrefix()} Validation error. Invalid file object. Specify a valid file object.`; }, + get INVALID_BASE64_STRING(): string { return `${errorPrefix()} Validation error. Invalid base64 string. Specify a valid base64 string.`; }, + get INVALID_DEIDENTIFY_FILE_OPTIONS(): string { return `${errorPrefix()} Validation error. Invalid deidentify file options. Specify a valid deidentify file options.`; }, + get INVALID_ENTITIES(): string { return `${errorPrefix()} Validation error. Invalid entities. Specify valid entities as string array.`; }, + get EMPTY_ENTITIES(): string { return `${errorPrefix()} Validation error. Entities cannot be empty. Specify valid entities.`; }, + get EMPTY_ALLOW_REGEX_LIST(): string { return `${errorPrefix()} Validation error. Allow regex list cannot be empty. Specify valid allow regex list.`; }, + get INVALID_ALLOW_REGEX(): string { return `${errorPrefix()} Validation error. Invalid allow regex. Specify valid allow regex at index %s1.`; }, + get EMPTY_RESTRICT_REGEX_LIST(): string { return `${errorPrefix()} Validation error. Restrict regex list cannot be empty. Specify valid restrict regex list.`; }, + get INVALID_RESTRICT_REGEX(): string { return `${errorPrefix()} Validation error. Invalid restrict regex. Specify valid restrict regex at index %s1.`; }, + get INVALID_OUTPUT_PROCESSED_IMAGE(): string { return `${errorPrefix()} Validation error. Invalid output processed image. Specify valid output processed image as string.`; }, + get INVALID_OUTPUT_OCR_TEXT(): string { return `${errorPrefix()} Validation error. Invalid output ocr text. Specify valid output ocr text as string.`; }, + get INVALID_MASKING_METHOD(): string { return `${errorPrefix()} Validation error. Invalid masking method. Specify valid masking method as string.`; }, + get INVALID_PIXEL_DENSITY(): string { return `${errorPrefix()} Validation error. Invalid pixel density. Specify valid pixel density as string.`; }, + get INVALID_OUTPUT_TRANSCRIPTION(): string { return `${errorPrefix()} Validation error. Invalid output transcription. Specify valid output transcription as string.`; }, + get INVALID_OUTPUT_PROCESSED_AUDIO(): string { return `${errorPrefix()} Validation error. Invalid output processed audio. Specify valid output processed audio as string.`; }, + get INVALID_MAX_RESOLUTION(): string { return `${errorPrefix()} Validation error. Invalid max resolution. Specify valid max resolution as string.`; }, + get INVALID_BLEEP(): string { return `${errorPrefix()} Validation error. Invalid bleep. Specify valid bleep as object.`; }, + get INVALID_FILE_OR_ENCODED_FILE(): string { return `${errorPrefix()} . Error while decoding base64 and saving file`; }, + get INVALID_FILE_TYPE(): string { return `${errorPrefix()} Validation error. Invalid file type. Specify a valid file type.`; }, + get INVALID_DEIDENTIFY_FILE_PATH(): string { return `${errorPrefix()} Validation error. Invalid file path. Specify a valid file path.`; }, + get INVALID_FILE_NAME(): string { return `${errorPrefix()} Validation error. Invalid file name. Specify a valid file name.`; }, + get FILE_READ_ERROR(): string { return `${errorPrefix()} Validation error. Unable to read file. Verify the file path.`; }, + get INVALID_BASE64_HEADER(): string { return `${errorPrefix()} Validation error. Invalid base64 header. Specify a valid base64 header.`; }, + get INVALID_WAIT_TIME(): string { return `${errorPrefix()} Validation error. Invalid wait time. Specify a valid wait time as number and should not be greater than 20 secs.`; }, + get INVALID_OUTPUT_DIRECTORY(): string { return `${errorPrefix()} Validation error. Invalid output directory. Specify a valid output directory as string.`; }, + get INVALID_OUTPUT_DIRECTORY_PATH(): string { return `${errorPrefix()} Validation error. Invalid output directory path. Specify a valid output directory path as string.`; }, + get EMPTY_RUN_ID(): string { return `${errorPrefix()} Validation error. Run id cannot be empty. Specify a valid run id.`; }, + get INVALID_RUN_ID(): string { return `${errorPrefix()} Validation error. Invalid run id. Specify a valid run id as string.`; }, + get INTERNAL_SERVER_ERROR(): string { return `${errorPrefix()}. Internal server error. %s1.`; }, + get INVALID_XML_FORMAT(): string { return `${errorPrefix()} Validation error. Invalid XML format. Specify a valid XML format as string.`; }, +}; + +export default errorMessages; \ No newline at end of file diff --git a/common/src/identity.ts b/common/src/identity.ts new file mode 100644 index 00000000..a9453317 --- /dev/null +++ b/common/src/identity.ts @@ -0,0 +1,15 @@ +export interface SdkIdentity { + name: string; + version: string; +} + +// Default identity = legacy product. Layers (skyvault/flowvault) override via setSdkIdentity(). +let current: SdkIdentity = { name: 'skyflow-node', version: '2.1.2' }; + +export function setSdkIdentity(id: SdkIdentity): void { + current = { ...id }; +} + +export function getSdkIdentity(): SdkIdentity { + return { ...current }; +} diff --git a/common/src/index.ts b/common/src/index.ts new file mode 100644 index 00000000..7f8065e3 --- /dev/null +++ b/common/src/index.ts @@ -0,0 +1,82 @@ +// ============================================================================= +// skyflow-common: contract-agnostic shared core for the Skyflow Node SDKs. +// Layers (skyvault/flowvault) consume this surface via `import { ... } from 'skyflow-common'`. +// ============================================================================= + +// --- Utils: enums, consts, helpers and shared interfaces --------------------- +// Exposes Env, LogLevel, RequestMethod, RedactionType, OrderByEnum, TokenMode, +// TokenType, MaskingMethod, DetectOutputTranscription, DetectEntities, MessageType, +// AuthType, CONFIG, CONTROLLER_TYPES, TYPES, SKYFLOW, SDK, HTTP_STATUS_CODE, +// CONTENT_TYPE, HTTP_HEADER, REQUEST (and other consts); printLog, parameterizedString, +// generateSDKMetrics, getVaultURL, getConnectionBaseURL, getBearerToken, getToken, +// validateToken, removeSDKVersion, getBaseUrl, fillUrlWithPathAndQueryParams, +// isValidURL, objectToXML; plus the ISkyflowError, SkyflowRecordError, AuthInfo types. +export * from "./utils"; + +// --- Errors ------------------------------------------------------------------ +export { default as SkyflowError } from "./error"; +export { default as SKYFLOW_ERROR_CODE } from "./error/codes"; +export { default as errorMessages } from "./error/messages"; +export { default as logs } from "./utils/logs"; + +// --- JWT helpers ------------------------------------------------------------- +export { isExpired, isTokenValid } from "./utils/jwt-utils"; + +// --- Credentials & config ---------------------------------------------------- +export { default as Credentials } from "./config/credentials"; +export type { + ApiKeyCredentials, + TokenCredentials, + PathCredentials, + StringCredentials, +} from "./config/credentials"; +export { default as VaultConfig } from "./config/vault"; +export { default as ConnectionConfig } from "./config/connection"; + +// --- Shared, contract-agnostic types ----------------------------------------- +// SkyflowConfig, StringKeyValueMapType, FileInput, Filepath, FileObject, +// InsertResponseType, GetResponseData, QueryResponseType, TokenizeRequestType, +// IndexRange, ServiceAccountResponseError. +export * from "./types"; + +// --- Service account (token generation) -------------------------------------- +export { + generateBearerToken, + generateBearerTokenFromCreds, + generateSignedDataTokens, + generateSignedDataTokensFromCreds, +} from "./service-account"; +export type { + BearerTokenOptions, + SignedDataTokensOptions, + GenerateTokenOptions, + SignedDataTokensResponse, + TokenResponse, +} from "./service-account"; + +// --- Generic validators ------------------------------------------------------ +export { + validateSkyflowConfig, + validateVaultConfig, + validateConnectionConfig, + validateUpdateVaultConfig, + validateUpdateConnectionConfig, + validateSkyflowCredentials, + validateCredentialsWithId, + isValidAPIKey, + isLogLevel, + isEnv, +} from "./validations"; + +// --- SDK identity ------------------------------------------------------------ +export { setSdkIdentity, getSdkIdentity } from "./identity"; +export type { SdkIdentity } from "./identity"; + +// --- Base (contract-agnostic) model classes ---------------------------------- +export { BaseOptions } from "./base/options"; +export * from "./base/request"; +export * from "./base/response"; + +// --- Facade base (shared Skyflow client scaffolding) ------------------------- +export { BaseSkyflow } from "./skyflow/base"; +export type { BaseVaultEntry, BaseVaultClientLike } from "./skyflow/base"; diff --git a/src/service-account/client/index.ts b/common/src/service-account/client/index.ts similarity index 69% rename from src/service-account/client/index.ts rename to common/src/service-account/client/index.ts index 834fca56..2c16cfdc 100644 --- a/src/service-account/client/index.ts +++ b/common/src/service-account/client/index.ts @@ -1,5 +1,5 @@ // imports -import { Authentication } from "../../ _generated_/rest/api/resources/authentication/client/Client"; +import { Authentication } from "../../_generated_/rest/api/resources/authentication/client/Client"; class Client { diff --git a/src/service-account/index.ts b/common/src/service-account/index.ts similarity index 99% rename from src/service-account/index.ts rename to common/src/service-account/index.ts index a68903e0..320884b8 100644 --- a/src/service-account/index.ts +++ b/common/src/service-account/index.ts @@ -1,13 +1,13 @@ import * as fs from 'fs'; import jwt from "jsonwebtoken"; -import { V1GetAuthTokenRequest, V1GetAuthTokenResponse } from '../ _generated_/rest/api'; +import { V1GetAuthTokenRequest, V1GetAuthTokenResponse } from '../_generated_/rest/api'; import { getBaseUrl, LogLevel, isValidURL, MessageType, parameterizedString, printLog, HTTP_HEADER, CONTENT_TYPE, HTTP_STATUS_CODE, JWT, ENCODING_TYPE } from '../utils'; import Client from './client'; import logs from '../utils/logs'; import SkyflowError from '../error'; import SKYFLOW_ERROR_CODE from '../error/codes'; -import { ServiceAccountResponseError } from '../vault/types'; -import { WithRawResponse } from '../ _generated_/rest/core'; +import { ServiceAccountResponseError } from '../types'; +import { WithRawResponse } from '../_generated_/rest/core'; function normalizeTokenOptions(options?: BearerTokenOptions): BearerTokenOptions | undefined { if (!options) return options; diff --git a/common/src/skyflow/base.ts b/common/src/skyflow/base.ts new file mode 100644 index 00000000..6d899164 --- /dev/null +++ b/common/src/skyflow/base.ts @@ -0,0 +1,219 @@ +// ============================================================================= +// BaseSkyflow — the contract-agnostic facade base shared by every Skyflow SDK +// (skyvault / flowvault), analogous to Java's `common` BaseSkyflow. +// +// It owns the vault-client registry, common credentials, log level, and the +// vault-config / credential / log-level management methods. Each layer extends +// it, implements `buildVaultEntry` to construct its own VaultClient + the +// controllers it exposes, and adds its layer-specific accessors (vault(), +// and for skyvault also detect()/connection()). +// ============================================================================= +import { + CONFIG, + Env, + LogLevel, + MessageType, + parameterizedString, + printLog, + getVaultURL, +} from "../utils"; +import { + isLogLevel, + validateSkyflowConfig, + validateVaultConfig, + validateUpdateVaultConfig, + validateSkyflowCredentials, +} from "../validations"; +import SkyflowError from "../error"; +import SKYFLOW_ERROR_CODE from "../error/codes"; +import logs from "../utils/logs"; +import type Credentials from "../config/credentials"; +import type VaultConfig from "../config/vault"; +import type { SkyflowConfig } from "../types"; + +/** Minimal surface each layer's VaultClient must expose for base management. */ +export interface BaseVaultClientLike { + setLogLevel(logLevel: LogLevel): void; + updateSkyflowCredentials(credentials?: Credentials): void; + updateClientConfig( + url: string, + vaultId: string, + individualCredentials?: Credentials, + skyflowCredentials?: Credentials, + logLevel?: LogLevel, + ): void; +} + +/** A registry entry for one configured vault: its config, client and controllers. */ +export interface BaseVaultEntry { + config: VaultConfig; + client: TClient; +} + +export abstract class BaseSkyflow { + protected vaultClients: { [vaultId: string]: TEntry } = {}; + + protected commonCredentials?: Credentials; + + protected logLevel: LogLevel = LogLevel.ERROR; + + /** + * Build the layer-specific entry (VaultClient + the controllers that layer + * exposes). Implementations may read `this.commonCredentials` / `this.logLevel`. + */ + protected abstract buildVaultEntry(config: VaultConfig): TEntry; + + /** + * Common initialization: validate config, set log level + credentials, and + * register the provided vault configs. Call from the subclass constructor + * (after super()). Connection configs, if any, are handled by the subclass. + */ + protected initializeSkyflow(config: SkyflowConfig): void { + validateSkyflowConfig(config, this.logLevel); + printLog(logs.infoLogs.INITIALIZE_CLIENT, MessageType.LOG, this.logLevel); + + if (config?.logLevel && !isLogLevel(config.logLevel)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_LOG_LEVEL); + } + this.logLevel = config.logLevel || LogLevel.ERROR; + printLog( + parameterizedString(logs.infoLogs.CURRENT_LOG_LEVEL, [this.logLevel]), + MessageType.LOG, + this.logLevel, + ); + + if (config?.skyflowCredentials) { + validateSkyflowCredentials(config.skyflowCredentials); + } + this.commonCredentials = config?.skyflowCredentials; + + printLog(logs.infoLogs.VALIDATING_VAULT_CONFIG, MessageType.LOG, this.logLevel); + config.vaultConfigs?.forEach((vaultConfig) => this.addVaultConfig(vaultConfig)); + } + + addVaultConfig(config: VaultConfig): void { + validateVaultConfig(config, this.logLevel); + this.throwErrorIfVaultIdExists(config?.vaultId); + this.vaultClients[config.vaultId] = this.buildVaultEntry(config); + printLog(logs.infoLogs.CONTROLLER_INITIALIZED, MessageType.LOG, this.logLevel); + } + + updateVaultConfig(config: VaultConfig): void { + validateUpdateVaultConfig(config, this.logLevel); + const existing = this.vaultClients[config.vaultId]; + if (!existing) { + this.throwErrorForUnknownVaultId(config.vaultId); + return; + } + const updatedConfig = { ...existing.config, ...config }; + const vaultUrl = getVaultURL(updatedConfig.clusterId, updatedConfig.env || Env.PROD); + existing.config = updatedConfig; + existing.client.updateClientConfig( + vaultUrl, + updatedConfig.vaultId, + updatedConfig.credentials, + this.commonCredentials, + this.logLevel, + ); + } + + getVaultConfig(vaultId: string): VaultConfig { + if (!vaultId) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_ID_VALIDATION); + } + const existing = this.vaultClients[vaultId]; + if (!existing) { + this.throwErrorForUnknownVaultId(vaultId); + } + return existing.config; + } + + removeVaultConfig(vaultId: string): void { + if (!this.vaultClients[vaultId]) { + this.throwErrorForUnknownVaultId(vaultId); + return; + } + delete this.vaultClients[vaultId]; + } + + setLogLevel(logLevel: LogLevel): void { + if (logLevel && !isLogLevel(logLevel)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_LOG_LEVEL); + } + this.logLevel = logLevel; + this.applyLogLevel(); + } + + /** Apply the current log level to all clients. Subclasses extend to cover + * additional (e.g. connection) clients, then call super.applyLogLevel(). */ + protected applyLogLevel(): void { + Object.values(this.vaultClients).forEach((entry) => entry.client.setLogLevel(this.logLevel)); + } + + updateLogLevel(logLevel: LogLevel): this { + this.setLogLevel(logLevel); + return this; + } + + getLogLevel(): LogLevel { + return this.logLevel; + } + + updateSkyflowCredentials(credentials: Credentials): void { + if (!credentials) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CREDENTIALS); + } + validateSkyflowCredentials(credentials); + this.commonCredentials = credentials; + this.applyCredentials(); + } + + /** Apply the current common credentials to all clients. Subclasses extend. */ + protected applyCredentials(): void { + Object.values(this.vaultClients).forEach((entry) => + entry.client.updateSkyflowCredentials(this.commonCredentials), + ); + } + + getSkyflowCredentials(): Credentials | undefined { + return this.commonCredentials; + } + + /** Resolve a vault entry by id (or the first registered), with the common + * empty-registry / unknown-id errors. Layer accessors (vault/detect) use this. */ + protected resolveVaultEntry(vaultId?: string): TEntry { + if (Object.keys(this.vaultClients).length === 0) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_CLIENTS); + } + const clientId = vaultId || Object.keys(this.vaultClients)[0]; + const entry = this.vaultClients[clientId]; + if (entry) { + return entry; + } + this.throwErrorForUnknownVaultId(clientId); + // Unreachable: throwErrorForUnknownVaultId always throws. + throw new SkyflowError(SKYFLOW_ERROR_CODE.INTERNAL_SERVER_ERROR); + } + + protected throwErrorIfVaultIdExists(id: string): void { + if (Object.keys(this.vaultClients).includes(id)) { + printLog( + parameterizedString(logs.infoLogs[`${CONFIG.VAULT_ID}_CONFIG_EXISTS`], [id]), + MessageType.ERROR, + this.logLevel, + ); + throw new SkyflowError(SKYFLOW_ERROR_CODE.VAULT_ID_EXITS_IN_CONFIG_LIST, [id]); + } + } + + protected throwErrorForUnknownVaultId(id: string): void { + printLog( + parameterizedString(logs.infoLogs[`${CONFIG.VAULT_ID}_CONFIG_DOES_NOT_EXIST`], [id]), + MessageType.ERROR, + this.logLevel, + ); + throw new SkyflowError(SKYFLOW_ERROR_CODE.VAULT_ID_NOT_IN_CONFIG_LIST, [id]); + } +} + +export default BaseSkyflow; diff --git a/common/src/types/index.ts b/common/src/types/index.ts new file mode 100644 index 00000000..1f381db2 --- /dev/null +++ b/common/src/types/index.ts @@ -0,0 +1,66 @@ +import { LogLevel } from "../utils"; +import ConnectionConfig from "../config/connection"; +import VaultConfig from "../config/vault"; +import Credentials from "../config/credentials"; + +export interface SkyflowConfig { + vaultConfigs?: VaultConfig[]; + connectionConfigs?: ConnectionConfig[]; + skyflowCredentials?: Credentials; + logLevel?: LogLevel; +} + +export type FileInput = + | Filepath + | FileObject + +export interface Filepath { + filePath: string; +} + +export interface FileObject { + file: File; +} + +export interface InsertResponseType { + skyflowId: string; + /** @deprecated Renamed to skyflowId. Will be removed in v3. */ + skyflow_id?: string; + [key: string]: unknown; +} + +export interface GetResponseData { + [key: string]: unknown; +} + +export interface QueryResponseType { + [key: string]: unknown; +} + +export interface StringKeyValueMapType { + [key: string]: object | string; +} + +export interface TokenizeRequestType { + columnGroup: string; + value: string; +} + +export interface IndexRange { + start?: number; + end?: number; +} + +export type ServiceAccountResponseError = { + rawResponse?: { + headers?: { + get: (header: string) => string | undefined; + }; + }; + body: any; + response?: { + status?: number | string; + }; + message: string; + [key: string]: any; +}; diff --git a/src/utils/index.ts b/common/src/utils/index.ts similarity index 97% rename from src/utils/index.ts rename to common/src/utils/index.ts index 983aac2e..606f3575 100644 --- a/src/utils/index.ts +++ b/common/src/utils/index.ts @@ -1,16 +1,16 @@ import SkyflowError from "../error"; -import * as sdkDetails from "../../package.json"; +import { getSdkIdentity } from "../identity"; import { generateBearerToken, generateBearerTokenFromCreds } from "../service-account"; import type { BearerTokenOptions } from "../service-account"; -import Credentials, { ApiKeyCredentials, PathCredentials, StringCredentials, TokenCredentials } from "../vault/config/credentials"; +import Credentials, { ApiKeyCredentials, PathCredentials, StringCredentials, TokenCredentials } from "../config/credentials"; import dotenv from "dotenv"; import logs from "./logs"; import os from 'os'; import process from "process"; import SKYFLOW_ERROR_CODE from "../error/codes"; import { isExpired } from "./jwt-utils"; -import { isValidAPIKey } from "./validations"; -import { StringKeyValueMapType } from "../vault/types"; +import { isValidAPIKey } from "../validations"; +import { StringKeyValueMapType } from "../types"; dotenv.config(); @@ -517,7 +517,7 @@ export const printLog = (message: string, messageType: MessageType, logLevel: Lo const { showDebugLogs, showInfoLogs, showWarnLogs, showErrorLogs, } = LogLevelOptions[logLevel]; - const version = sdkDetails.version ? `v${sdkDetails.version}` : ''; + const version = getSdkIdentity().version ? `v${getSdkIdentity().version}` : ''; if (messageType === MessageType.LOG && showDebugLogs) { // eslint-disable-next-line no-console console.log(`DEBUG: [Skyflow Node SDK ${version}] ` + message); @@ -548,7 +548,7 @@ export const generateSDKMetrics = (logLevel?: LogLevel) => { let clientOSDetails = ""; let runtimeDetails = ""; try { - sdkNameVersion = `${sdkDetails.name ? `${sdkDetails.name}@` : ""}${sdkDetails.version ? sdkDetails.version : "" + sdkNameVersion = `${getSdkIdentity().name ? `${getSdkIdentity().name}@` : ""}${getSdkIdentity().version ? getSdkIdentity().version : "" }`; } catch (err) { printLog( diff --git a/src/utils/jwt-utils/index.ts b/common/src/utils/jwt-utils/index.ts similarity index 100% rename from src/utils/jwt-utils/index.ts rename to common/src/utils/jwt-utils/index.ts diff --git a/src/utils/logs/index.ts b/common/src/utils/logs/index.ts similarity index 100% rename from src/utils/logs/index.ts rename to common/src/utils/logs/index.ts diff --git a/common/src/validations/index.ts b/common/src/validations/index.ts new file mode 100644 index 00000000..c3fd4498 --- /dev/null +++ b/common/src/validations/index.ts @@ -0,0 +1,380 @@ +import { CONFIG, Env, isValidURL, LogLevel, MessageType, printLog, API_KEY } from "../utils"; +import SkyflowError from "../error"; +import SKYFLOW_ERROR_CODE from "../error/codes"; +import ConnectionConfig from "../config/connection"; +import Credentials, { ApiKeyCredentials, PathCredentials, StringCredentials, TokenCredentials } from "../config/credentials"; +import VaultConfig from "../config/vault"; +import { SkyflowConfig } from "../types"; +import * as fs from 'fs'; +import { isExpired } from "../utils/jwt-utils"; +import logs from "../utils/logs"; + +export function isEnv(value?: string): boolean { + return value !== undefined && Object.values(Env).includes(value as Env); +} + +export function isLogLevel(value?: string): boolean { + return value !== undefined && Object.values(LogLevel).includes(value as LogLevel); +} + +export function isValidAPIKey(apiKey: string) { + if (!apiKey) { + return false; + } + if (apiKey && typeof apiKey === 'string' && apiKey.startsWith(API_KEY.PREFIX)) { + return true; + } + return false; +}; + +function isValidCredentialsString(credentialsString: string) { + if (!credentialsString || credentialsString === null || credentialsString === undefined) { + return false; + } + if (credentialsString && typeof credentialsString === 'string') { + try { + let credentialsObj = JSON.parse("{}") + const parsed = JSON.parse(credentialsString); + credentialsObj = { + ...parsed, + clientId: parsed.clientId ?? parsed.clientID, + keyId: parsed.keyId ?? parsed.keyID, + tokenUri: parsed.tokenUri ?? parsed.tokenURI, + }; + if (credentialsObj?.clientId == null || credentialsObj?.keyId == null) { + return false; + } + return true; + } catch (err) { + return false; + } + } + return false; +}; + +function isValidPath(path: string) { + if (!path || path === null || path === undefined) { + return false; + } + if (path && typeof path === 'string' && fs.existsSync(path)) { + return true; + } + return false; +}; + +export const validateSkyflowConfig = (config: SkyflowConfig, logLevel: LogLevel = LogLevel.ERROR) => { + if (config) { + if (!Object.prototype.hasOwnProperty.call(config, 'vaultConfigs')) { + printLog(logs.errorLogs.VAULT_CONFIG_KEY_MISSING, MessageType.ERROR, logLevel); + } + + const { vaultConfigs, connectionConfigs } = config; + + // Count how many of the fields are defined + const definedFields = [vaultConfigs, connectionConfigs].filter(Boolean).length; + + // If none are present + if (definedFields === 0) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_SKYFLOW_CONFIG); + } + + if (config?.vaultConfigs && !Array.isArray(config.vaultConfigs)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TYPE_FOR_CONFIG, [CONFIG.VAULT]) + } + + if (config?.connectionConfigs && !Array.isArray(config?.connectionConfigs)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TYPE_FOR_CONFIG, [CONFIG.CONNECTION]) + } + + } else { + throw new SkyflowError(SKYFLOW_ERROR_CODE.CONFIG_MISSING); + } +}; + + +export const validateCredentialsWithId = (credentials: Credentials, type: string, typeId: string, id: string, logLevel: LogLevel = LogLevel.ERROR) => { + if (!credentials) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CREDENTIALS_WITH_ID, [type, typeId, id]); + } + + const isTokenCred = 'token' in credentials; + const isPathCred = 'path' in credentials; + const isStringCred = 'credentialsString' in credentials; + const isApiKeyCred = 'apiKey' in credentials; + + // Check if exactly one credential type is provided + const definedTypes = [isTokenCred, isPathCred, isStringCred, isApiKeyCred].filter(Boolean).length; + + if (definedTypes === 0) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CREDENTIALS_WITH_ID, [type, typeId, id]); + } + + if (definedTypes > 1) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.MULTIPLE_CREDENTIALS_PASSED_WITH_ID, [type, typeId, id]); + } + + // Validate TokenCredentials + if (isTokenCred) { + const tokenCred = credentials as TokenCredentials; + if (typeof tokenCred.token !== 'string' || isExpired(tokenCred.token)) { + printLog(logs.errorLogs.EMPTY_TOKEN_VALUE, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_BEARER_TOKEN_WITH_ID, [type, typeId, id]); + } + } + + // Validate PathCredentials + if (isPathCred) { + const pathCred = credentials as PathCredentials; + if (typeof pathCred.path !== 'string' || !isValidPath(pathCred.path)) { + printLog(logs.errorLogs.EMPTY_CREDENTIALS_PATH, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_FILE_PATH_WITH_ID, [type, typeId, id]); + } + if (pathCred.roles !== undefined && !Array.isArray(pathCred.roles)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE, [type, typeId, id]); + } + if (pathCred.context !== undefined && (typeof pathCred.context !== 'string' && typeof pathCred.context !== 'object')) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT, [type, typeId, id]); + } + if(Object.prototype.hasOwnProperty.call(pathCred, 'tokenUri')) { + if (pathCred.tokenUri === undefined || typeof pathCred.tokenUri !== 'string' || !isValidURL(pathCred.tokenUri)) { + printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI_WITH_ID, [type, typeId, id]); + } + } + } + + // Validate StringCredentials + if (isStringCred) { + const stringCred = credentials as StringCredentials; + if (typeof stringCred.credentialsString !== 'string' || !isValidCredentialsString(stringCred.credentialsString)) { + printLog(logs.errorLogs.EMPTY_CREDENTIALS_STRING, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_PARSED_CREDENTIALS_STRING_WITH_ID, [type, typeId, id]); + } + if (stringCred.roles !== undefined && !Array.isArray(stringCred.roles)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE, [type, typeId, id]); + } + if (stringCred.context !== undefined && (typeof stringCred.context !== 'string' && typeof stringCred.context !== 'object')) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT, [type, typeId, id]); + } + if (Object.prototype.hasOwnProperty.call(stringCred, 'tokenUri')) { + if (stringCred.tokenUri === undefined || typeof stringCred.tokenUri !== 'string' || !isValidURL(stringCred.tokenUri)) { + printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI_WITH_ID, [type, typeId, id]); + } + } + } + + // Validate ApiKeyCredentials + if (isApiKeyCred) { + const apiKeyCred = credentials as ApiKeyCredentials; + if (typeof apiKeyCred.apiKey !== 'string' || !isValidAPIKey(apiKeyCred.apiKey)) { + printLog(logs.errorLogs.INVALID_KEY, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_KEY_WITH_ID, [type, typeId, id]); + } + } +}; + +export const validateVaultConfig = (vaultConfig: VaultConfig, logLevel: LogLevel = LogLevel.ERROR) => { + if (vaultConfig) { + if (!Object.prototype.hasOwnProperty.call(vaultConfig, 'vaultId')) { + printLog(logs.errorLogs.EMPTY_VAULT_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_ID); + } + if (!vaultConfig.vaultId || typeof vaultConfig.vaultId !== 'string') { + printLog(logs.errorLogs.INVALID_VAULT_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_VAULT_ID); + } + if (!Object.prototype.hasOwnProperty.call(vaultConfig, 'clusterId')) { + printLog(logs.errorLogs.EMPTY_CLUSTER_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CLUSTER_ID, [vaultConfig?.vaultId]); + } + if (!vaultConfig.clusterId || typeof vaultConfig.clusterId !== 'string') { + printLog(logs.errorLogs.INVALID_CLUSTER_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CLUSTER_ID, [vaultConfig?.vaultId]); + } + if (vaultConfig?.env && !isEnv(vaultConfig.env)) { + printLog(logs.errorLogs.INVALID_ENV, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ENV, [vaultConfig?.vaultId]); + } + if (vaultConfig?.credentials) { + validateCredentialsWithId(vaultConfig.credentials, CONFIG.VAULT, CONFIG.VAULT_ID, vaultConfig.vaultId, logLevel); + } + } else { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_CONFIG); + } +}; + +export const validateUpdateVaultConfig = (vaultConfig: VaultConfig, logLevel: LogLevel = LogLevel.ERROR) => { + if (vaultConfig) { + if (!Object.prototype.hasOwnProperty.call(vaultConfig, 'vaultId')) { + printLog(logs.errorLogs.EMPTY_VAULT_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_ID); + } + if (!vaultConfig?.vaultId || typeof vaultConfig?.vaultId !== 'string') { + printLog(logs.errorLogs.INVALID_VAULT_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_VAULT_ID); + } + if (vaultConfig?.clusterId && typeof vaultConfig.clusterId !== 'string') { + printLog(logs.errorLogs.INVALID_CLUSTER_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CLUSTER_ID, [vaultConfig?.vaultId]); + } + if (vaultConfig?.env && !isEnv(vaultConfig.env)) { + printLog(logs.errorLogs.INVALID_ENV, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ENV, [vaultConfig?.vaultId]); + } + if (vaultConfig?.credentials) { + validateCredentialsWithId(vaultConfig.credentials, CONFIG.VAULT, CONFIG.VAULT_ID, vaultConfig.vaultId, logLevel); + } + } else { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_CONFIG); + } +}; + +export const validateSkyflowCredentials = (credentials: Credentials, logLevel: LogLevel = LogLevel.ERROR) => { + if (!credentials) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.CREDENTIALS_WITH_NO_VALID_KEY); + } + + const isTokenCred = 'token' in credentials; + const isPathCred = 'path' in credentials; + const isStringCred = 'credentialsString' in credentials; + const isApiKeyCred = 'apiKey' in credentials; + + // Check if exactly one credential type is provided + const definedTypes = [isTokenCred, isPathCred, isStringCred, isApiKeyCred].filter(Boolean).length; + + if (definedTypes === 0) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.CREDENTIALS_WITH_NO_VALID_KEY); + } + + if (definedTypes > 1) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.MULTIPLE_CREDENTIALS_PASSED); + } + + // Validate TokenCredentials + if (isTokenCred) { + const tokenCred = credentials as TokenCredentials; + if (typeof tokenCred.token !== 'string' || isExpired(tokenCred.token)) { + printLog(logs.errorLogs.EMPTY_TOKEN_VALUE, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_BEARER_TOKEN); + } + } + + // Validate PathCredentials + if (isPathCred) { + const pathCred = credentials as PathCredentials; + if (typeof pathCred.path !== 'string' || !isValidPath(pathCred.path)) { + printLog(logs.errorLogs.EMPTY_CREDENTIALS_PATH, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CREDENTIALS_FILE_PATH); + } + if (pathCred.roles !== undefined && !Array.isArray(pathCred.roles)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE); + } + if (pathCred.context !== undefined && (typeof pathCred.context !== 'string' && typeof pathCred.context !== 'object')) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT); + } + + if(Object.prototype.hasOwnProperty.call(pathCred, 'tokenUri')) { + if (pathCred.tokenUri === undefined || typeof pathCred.tokenUri !== 'string' || !isValidURL(pathCred.tokenUri)) { + printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI); + } + } + } + + // Validate StringCredentials + if (isStringCred) { + const stringCred = credentials as StringCredentials; + if (typeof stringCred.credentialsString !== 'string' || !isValidCredentialsString(stringCred.credentialsString)) { + printLog(logs.errorLogs.EMPTY_CREDENTIALS_STRING, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_PARSED_CREDENTIALS_STRING); + } + if (stringCred.roles !== undefined && !Array.isArray(stringCred.roles)) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE); + } + // validate both string | Record + if (stringCred.context !== undefined && (typeof stringCred.context !== 'string' && typeof stringCred.context !== 'object')) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT); + } + if (Object.prototype.hasOwnProperty.call(stringCred, 'tokenUri')) { + if (stringCred.tokenUri === undefined || typeof stringCred.tokenUri !== 'string' || !isValidURL(stringCred.tokenUri)) { + printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI); + } + } + } + + // Validate ApiKeyCredentials + if (isApiKeyCred) { + const apiKeyCred = credentials as ApiKeyCredentials; + if (typeof apiKeyCred.apiKey !== 'string' || !isValidAPIKey(apiKeyCred.apiKey)) { + printLog(logs.errorLogs.INVALID_KEY, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_KEY); + } + } +}; + +export const validateConnectionConfig = (connectionConfig: ConnectionConfig, logLevel: LogLevel = LogLevel.ERROR) => { + if (connectionConfig) { + if (!Object.prototype.hasOwnProperty.call(connectionConfig, 'connectionId')) { + printLog(logs.errorLogs.EMPTY_CONNECTION_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_ID); + } + + if (typeof connectionConfig?.connectionId !== 'string' || connectionConfig?.connectionId.trim().length === 0) { + printLog(logs.errorLogs.INVALID_CONNECTION_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_ID); + } + + if (!Object.prototype.hasOwnProperty.call(connectionConfig, 'connectionUrl')) { + printLog(logs.errorLogs.EMPTY_CONNECTION_URL, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_URL); + } + + if (typeof connectionConfig?.connectionUrl !== 'string' || connectionConfig?.connectionUrl.trim().length === 0) { + printLog(logs.errorLogs.INVALID_CONNECTION_URL, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); + } + + if (connectionConfig.connectionUrl && !isValidURL(connectionConfig.connectionUrl)) { + printLog(logs.errorLogs.INVALID_CONNECTION_URL_TYPE, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); + } + + if (connectionConfig?.credentials) { + validateCredentialsWithId(connectionConfig.credentials, CONFIG.CONNECTION, CONFIG.CONNECTION_ID, connectionConfig.connectionId, logLevel); + } + } else { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_CONFIG) + } +}; + +export const validateUpdateConnectionConfig = (connectionConfig: ConnectionConfig, logLevel: LogLevel = LogLevel.ERROR) => { + if (connectionConfig) { + if (!Object.prototype.hasOwnProperty.call(connectionConfig, 'connectionId')) { + printLog(logs.errorLogs.EMPTY_CONNECTION_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_ID); + } + + if (typeof connectionConfig?.connectionId !== 'string' || connectionConfig?.connectionId.trim().length === 0) { + printLog(logs.errorLogs.INVALID_CONNECTION_ID, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_ID); + } + + if (connectionConfig?.connectionUrl && (typeof connectionConfig?.connectionUrl !== 'string' || connectionConfig?.connectionUrl.trim().length === 0)) { + printLog(logs.errorLogs.INVALID_CONNECTION_URL, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); + } + + if (connectionConfig?.connectionUrl && !isValidURL(connectionConfig.connectionUrl)) { + printLog(logs.errorLogs.INVALID_CONNECTION_URL_TYPE, MessageType.ERROR, logLevel); + throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); + } + + if (connectionConfig?.credentials) { + validateCredentialsWithId(connectionConfig.credentials, CONFIG.CONNECTION, CONFIG.CONNECTION_ID, connectionConfig.connectionId, logLevel); + } + } else { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_CONFIG) + } +}; diff --git a/test/demo-credentials/empty.json b/common/test/demo-credentials/empty.json similarity index 100% rename from test/demo-credentials/empty.json rename to common/test/demo-credentials/empty.json diff --git a/test/demo-credentials/invalidJson.json b/common/test/demo-credentials/invalidJson.json similarity index 100% rename from test/demo-credentials/invalidJson.json rename to common/test/demo-credentials/invalidJson.json diff --git a/test/demo-credentials/noClientId.json b/common/test/demo-credentials/noClientId.json similarity index 100% rename from test/demo-credentials/noClientId.json rename to common/test/demo-credentials/noClientId.json diff --git a/test/demo-credentials/noKeyId.json b/common/test/demo-credentials/noKeyId.json similarity index 100% rename from test/demo-credentials/noKeyId.json rename to common/test/demo-credentials/noKeyId.json diff --git a/test/demo-credentials/noPrivateKey.json b/common/test/demo-credentials/noPrivateKey.json similarity index 100% rename from test/demo-credentials/noPrivateKey.json rename to common/test/demo-credentials/noPrivateKey.json diff --git a/test/demo-credentials/noTokenURI.json b/common/test/demo-credentials/noTokenURI.json similarity index 100% rename from test/demo-credentials/noTokenURI.json rename to common/test/demo-credentials/noTokenURI.json diff --git a/test/demo-credentials/valid.json b/common/test/demo-credentials/valid.json similarity index 100% rename from test/demo-credentials/valid.json rename to common/test/demo-credentials/valid.json diff --git a/test/error/skyflow-error.test.js b/common/test/error/skyflow-error.test.js similarity index 100% rename from test/error/skyflow-error.test.js rename to common/test/error/skyflow-error.test.js diff --git a/test/service-account/client.test.js b/common/test/service-account/client.test.js similarity index 78% rename from test/service-account/client.test.js rename to common/test/service-account/client.test.js index 814b4cf2..5c8fbf37 100644 --- a/test/service-account/client.test.js +++ b/common/test/service-account/client.test.js @@ -1,11 +1,11 @@ -jest.mock('../../src/ _generated_/rest/api/resources/authentication/client/Client', () => ({ +jest.mock('../../src/_generated_/rest/api/resources/authentication/client/Client', () => ({ Authentication: jest.fn().mockImplementation((config) => ({ config })), })); const Client = require('../../src/service-account/client').default; const { Authentication, -} = require('../../src/ _generated_/rest/api/resources/authentication/client/Client'); +} = require('../../src/_generated_/rest/api/resources/authentication/client/Client'); describe('Service Account Client', () => { beforeEach(() => { diff --git a/test/service-account/token.test.js b/common/test/service-account/token.test.js similarity index 100% rename from test/service-account/token.test.js rename to common/test/service-account/token.test.js diff --git a/test/vault/utils/jwt-utils/jwt.test.js b/common/test/utils/jwt.test.js similarity index 93% rename from test/vault/utils/jwt-utils/jwt.test.js rename to common/test/utils/jwt.test.js index 74e16598..e1c560be 100644 --- a/test/vault/utils/jwt-utils/jwt.test.js +++ b/common/test/utils/jwt.test.js @@ -1,5 +1,5 @@ import { jwtDecode } from 'jwt-decode'; -import { isTokenValid, isExpired } from '../../../../src/utils/jwt-utils'; +import { isTokenValid, isExpired } from '../../src/utils/jwt-utils'; jest.mock('jwt-decode'); diff --git a/test/vault/utils/utils.test.js b/common/test/utils/utils.test.js similarity index 96% rename from test/vault/utils/utils.test.js rename to common/test/utils/utils.test.js index d7887d66..aaa51926 100644 --- a/test/vault/utils/utils.test.js +++ b/common/test/utils/utils.test.js @@ -1,14 +1,17 @@ /* eslint-disable camelcase */ -import errorMessages from "../../../src/error/messages"; -import { Env, getConnectionBaseURL, getVaultURL, validateToken, isValidURL, fillUrlWithPathAndQueryParams, generateSDKMetrics, printLog, getToken, getBearerToken, getBaseUrl, removeSDKVersion, parameterizedString, MessageType, LogLevel, objectToXML } from "../../../src/utils"; +import errorMessages from "../../src/error/messages"; +import { Env, getConnectionBaseURL, getVaultURL, validateToken, isValidURL, fillUrlWithPathAndQueryParams, generateSDKMetrics, printLog, getToken, getBearerToken, getBaseUrl, removeSDKVersion, parameterizedString, MessageType, LogLevel, objectToXML } from "../../src/utils"; import { jwtDecode as jwt_decode } from 'jwt-decode'; import os from 'os'; -import { generateBearerTokenFromCreds, generateBearerToken } from '../../../src/service-account'; -import sdkDetails from '../../../package.json'; +import { generateBearerTokenFromCreds, generateBearerToken } from '../../src/service-account'; +import sdkDetails from '../../../skyvault/package.json'; +// Same identity module instance that src/utils closes over (loaded above), so +// spying on it affects generateSDKMetrics/printLog even across jest.resetModules(). +const identityModule = require('../../src/identity'); jest.mock('jwt-decode'); -jest.mock('../../../src/service-account', () => ({ +jest.mock('../../src/service-account', () => ({ generateBearerTokenFromCreds: jest.fn(), generateBearerToken: jest.fn(), })); @@ -303,17 +306,19 @@ describe('generateSDKMetrics With errors', () => { Object.defineProperty(process, 'arch', { value: originalArch }); }); - test('should log error and set sdkNameVersion to empty if accessing sdkDetails throws an error', () => { - // Override the sdkDetails module to throw an error - jest.resetModules(); - Object.defineProperty(sdkDetails, 'name', { - get: () => { throw new Error('mock error'); } + test('should log error and set sdkNameVersion to empty if accessing the SDK identity throws an error', () => { + // In skyflow-common the name/version come from getSdkIdentity(), not package.json. + // Make accessing `.name` throw so generateSDKMetrics falls into its catch block. + jest.spyOn(identityModule, 'getSdkIdentity').mockReturnValue({ + get name() { throw new Error('mock error'); }, + version: '2.1.2', }); const metrics = generateSDKMetrics(logLevel); // Ensure sdkNameVersion is set to an empty string expect(metrics.sdk_name_version).toBe(""); + jest.restoreAllMocks(); }); }); @@ -578,13 +583,11 @@ describe('printLog version fallback', () => { jest.restoreAllMocks(); }); - test('uses empty version when sdkDetails.version is undefined', () => { - const origDescriptor = Object.getOwnPropertyDescriptor(sdkDetails, 'version'); - Object.defineProperty(sdkDetails, 'version', { value: undefined, writable: true, configurable: true }); + test('uses empty version when the SDK identity version is undefined', () => { + jest.spyOn(identityModule, 'getSdkIdentity').mockReturnValue({ name: 'skyflow-node', version: undefined }); const consoleSpy = jest.spyOn(console, 'log').mockImplementation(() => {}); printLog('test msg', MessageType.LOG, LogLevel.DEBUG); expect(consoleSpy).toHaveBeenCalledWith('DEBUG: [Skyflow Node SDK ] test msg'); - Object.defineProperty(sdkDetails, 'version', origDescriptor); }); }); @@ -593,15 +596,10 @@ describe('generateSDKMetrics branch coverage', () => { jest.restoreAllMocks(); }); - test('uses empty strings when sdkDetails name and version are falsy', () => { - const origName = Object.getOwnPropertyDescriptor(sdkDetails, 'name'); - const origVersion = Object.getOwnPropertyDescriptor(sdkDetails, 'version'); - Object.defineProperty(sdkDetails, 'name', { value: '', writable: true, configurable: true }); - Object.defineProperty(sdkDetails, 'version', { value: '', writable: true, configurable: true }); + test('uses empty strings when the SDK identity name and version are falsy', () => { + jest.spyOn(identityModule, 'getSdkIdentity').mockReturnValue({ name: '', version: '' }); const metrics = generateSDKMetrics(); expect(metrics.sdk_name_version).toBe(''); - Object.defineProperty(sdkDetails, 'name', origName); - Object.defineProperty(sdkDetails, 'version', origVersion); }); test('uses empty string for clientDeviceModel when process.platform is undefined', () => { @@ -626,7 +624,7 @@ describe('generateSDKMetrics branch coverage', () => { }); describe('objectToXML', () => { - const { objectToXML } = require('../../../src/utils'); + const { objectToXML } = require('../../src/utils'); test('should convert simple object to XML with default root', () => { const obj = { name: 'John', age: 30 }; @@ -897,7 +895,7 @@ describe('objectToXML', () => { }); describe('objectToXML', () => { - const { objectToXML } = require('../../../src/utils'); + const { objectToXML } = require('../../src/utils'); test('should convert simple object to XML with default root', () => { const obj = { name: 'John', age: 30 }; diff --git a/common/tsconfig.json b/common/tsconfig.json new file mode 100644 index 00000000..1d272823 --- /dev/null +++ b/common/tsconfig.json @@ -0,0 +1,34 @@ +{ + "compilerOptions": { + "composite": true, + "sourceMap": true, + "declaration": true, + "declarationMap": true, + "stripInternal": true, + "noImplicitAny": false, + "target": "es6", + "lib": ["es2015", "es2017", "es2019", "dom"], + "esModuleInterop": true, + "module": "commonjs", + "moduleResolution": "node", + "removeComments": false, + "allowSyntheticDefaultImports": true, + "allowJs": true, + "strict": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": false, + "isolatedModules": false, + "noEmit": false, + "rootDir": "src", + "outDir": "lib", + "types": ["node"] + }, + "include": ["src/**/*.ts"], + "exclude": [ + "node_modules", + "lib", + "src/_generated_/rest/Client.ts", + "src/_generated_/rest/index.ts" + ] +} diff --git a/docs/sdk-split-plan.md b/docs/sdk-split-plan.md new file mode 100644 index 00000000..20b3584d --- /dev/null +++ b/docs/sdk-split-plan.md @@ -0,0 +1,186 @@ +# Skyflow Node SDK — Split Architecture (`common` / `skyvault` / `flowvault`) + +## Context + +This repository historically shipped a single npm package, `skyflow-node`. To support a second +backend contract (**FlowVault**, v2) as its own installable SDK without forking the shared +plumbing, the repo is restructured into an **npm-workspaces monorepo** with three modules — +`common`, `skyvault`, `flowvault` — mirroring the Skyflow **Java** SDK split +(`common` / `skyvault` / `flowvault`). + +Which SDK an application gets is decided **solely by which npm package it installs** +(`npm i skyflow-node` vs `npm i skyflow-flowvault-node`): no build flags, no runtime mode +switches, no environment configuration. + +Design contract for the FlowVault surface: + + +## Module layout + +``` +/ (workspace root — private, not published) +├── common/ → package "skyflow-common" (private/internal; bundled into each published SDK) +├── skyvault/ → package "skyflow-node" (2.x; the original SDK, backward compatible) +├── flowvault/ → package "skyflow-flowvault-node" (1.x; the new v2 SDK) +└── package.json (workspaces: [common, skyvault, flowvault]; shared devDeps; build/test fan-out) +``` + +**Dependency direction:** `skyvault → common ← flowvault`. `common` depends on neither layer and +compiles & tests with neither present. A layer never imports the other layer. + +### `common` (contract-agnostic core) +Everything that is not tied to a specific wire contract: +- `error/` — `SkyflowError`, error codes, message catalog. +- `utils/` — enums/consts, logging, SDK metrics, URL + auth helpers, JWT helpers. +- `service-account/` — bearer-token / signed-data-token lifecycle (wired to the shared **auth** + generated client in `common/src/_generated_/`). +- `config/` — `Credentials` union, `VaultConfig`, `ConnectionConfig`. +- `validations/` — generic config/credential validators only. +- `types/` — pure, contract-agnostic types. +- `identity.ts` — the **mutable SDK identity** (`setSdkIdentity` / `getSdkIdentity`), default + `skyflow-node`. +- `base/` — `BaseOptions` + `BaseRequest` / `BaseResponse` marker classes (shared + extension points). +- `skyflow/base.ts` — **`BaseSkyflow`**, the shared facade base (vault-client registry, common + credentials, log level, and the `addVaultConfig`/`updateVaultConfig`/`getVaultConfig`/ + `removeVaultConfig`/`setLogLevel`/`updateLogLevel`/`getLogLevel`/`updateSkyflowCredentials`/ + `getSkyflowCredentials` methods). Each layer extends it, implements `buildVaultEntry` to construct + its own `VaultClient` + controllers, and adds its layer-specific accessors. Mirrors Java's + `common` `BaseSkyflow`. + +### `skyvault` (→ `skyflow-node`) +The original contract, unchanged behavior: vault data plane (insert/get/update/delete/ +detokenize/tokenize/query/file-upload) + connections + detect. Its `Skyflow` **extends +`BaseSkyflow`** (adding the detect controller per vault + the connection registry/`connection()`); +its `VaultClient`, controllers and models import shared code from `skyflow-common`; its concrete +model classes extend the `common` `Base*` classes; the per-request validators live here. It keeps +its existing data-plane generated client under `skyvault/src/ _generated_`. + +### `flowvault` (→ `skyflow-flowvault-node`) +The new v2 contract (unary only): insert/get/update/delete/query/detokenize/getTokens/uploadFiles/ +deleteFiles. Its `Skyflow` **extends `BaseSkyflow`** (just `buildVaultEntry` + `vault()`); its +models/options extend the same `Base*` classes; it wires to its own generated client at +`flowvault/_generated_/rest/src` (`.records`/`.tokens`/`.files`/`.query`). No connections/detect. + +## Consumer experience (one import per SDK) + +Each layer **re-exports `common`'s public surface** from its own `index.ts`, so applications +install exactly one package and import everything from it — they never import `skyflow-common` +directly and never install both SDKs. `skyflow-common` is an **unpublished** workspace package +**bundled into each published tarball** at build time; `/** @internal */` + `stripInternal` keep +internal symbols out of the published `.d.ts`, and only the public entry is exposed via each +package's `"exports"` map. + +## SDK identity (self-reporting) + +`common` holds a mutable identity object. Each layer **stamps** it at client construction: +- `skyvault` → `{ name: "skyflow-node", version: }` (also the default). +- `flowvault` → `setSdkIdentity({ name: "skyflow-flowvault-node", version: })`. + +Telemetry (`generateSDKMetrics` → `sky-metadata` header, `sdk_name_version`) reads the live +identity, so each SDK self-reports its own `name@version`. + +## Independent versioning & tagging + +Each published package has its own `version` and release tags: + +| Package | Version line | Git tag that triggers its release | +| --- | --- | --- | +| `skyflow-node` | 2.x (unchanged from today) | **plain semver** `x.y.z` (and `x.y.z-beta.n`) | +| `skyflow-flowvault-node` | 1.x (new) | **prefixed** `flowvault/x.y.z` (and `flowvault/x.y.z-beta.n`) | + +npm resolves packages by name from the registry (not from git tags), so the prefixed +`flowvault/*` tags are purely CI triggers — no distribution mirror repo is needed. The products +are distinguished solely by package name. + +### Version-bump script +`scripts/bump_version.sh [sha]` where `` is `skyflow-node` or +`skyflow-flowvault-node`; it bumps the right package's `package.json` `version` (and, for internal +builds, appends `-dev.`). + +## Release flow (CI) + +- **PR / push CI**: one command builds and tests **both** SDKs — the root `npm run build` + (`tsc -b` across the workspace) + `npm test` (fans out to each package). +- **Public release**: + - plain-semver tag → publish **`skyflow-node`** to npm only. + - `flowvault/*` tag → publish **`skyflow-flowvault-node`** to npm only. +- **Beta**: `x.y.z-beta.n` → `skyflow-node@beta`; `flowvault/x.y.z-beta.n` → + `skyflow-flowvault-node@beta`. +- **Internal (JFrog)**: branch-namespaced builds publish `-dev.` to the private registry, + per product. + +## Build & packaging + +- **Dev/CI build** — TypeScript **project references**: the root `npm run build` (`tsc -b`) builds + `common` first, then `skyvault` and `flowvault` against `common`'s declarations (each to its own + `lib/`). `npm test` fans out across the three workspaces. +- **Publish build** — each publishable package has a **tsup** bundle (`npm run build:dist`) that + **inlines `skyflow-common`** (`noExternal: ['skyflow-common']`) into a self-contained + `dist/index.js` + bundled `dist/index.d.ts`, keeping real npm deps external. `skyflow-common` is a + **devDependency** of each layer, so it never appears in the published `dependencies` (verified via + `npm pack`). `stripInternal` keeps `/** @internal */` symbols out of the published `.d.ts`, and + each package's `"exports"` map exposes only `.` (CJS `require` + `types`). Common's runtime deps + (`jsonwebtoken`, `jwt-decode`) are declared in each layer's `dependencies` since common is inlined. +- **Verified**: `npm pack` → install the tarball into a fresh consumer → `require()` works for both + SDKs; skyvault keeps all 74 public type names and the `v2.1.2` prefix, flowvault self-reports + `v1.0.0`. + +### Minor packaging note +The bundler materializes a handful of **type-only** exports (`Credentials`, `VaultConfig`, +`ConnectionConfig`, the credential subtypes, the service-account option types, `SkyflowRecordError`) +as inert `undefined` runtime keys on skyvault's CJS export object — because their source re-exports +use value syntax (`export { X }`) rather than `export type { X }`. This is harmless (a type has no +runtime value) and the full type surface is preserved in the `.d.ts`; marking those re-exports +`export type` in `common`'s barrel + the layer barrels would remove the extra keys. + +## Generated code (maintainer-owned) + +The OpenAPI spec is split and the client/transport code is generated **per module**: +`common/src/_generated_/` (auth), `skyvault/src/ _generated_` (v1 data plane), +`flowvault/_generated_/rest/src` (v2 data plane). Generated directories are treated as **provided +inputs** — not generated, moved, renamed, or hand-edited by the split. Where the hand-written +layer needs something the generated surface doesn't provide, it is **flagged** for the maintainer +to regenerate (see "Known flags" below). + +## Known flags (for the maintainer) + +1. **common auth generated `Client.ts` fails `strict`** — `common/src/_generated_/rest/Client.ts` + (a management-plane `patchV2VaultsId` stub) calls `encodeURIComponent(vaultId)` on a possibly + `undefined` `vaultId` (TS2345). It is unused by `common` (service-account uses the + `authentication` resource directly), so it + `rest/index.ts` are `exclude`d in + `common/tsconfig.json`. To compile them, the generated code needs a null-guard — a regeneration + concern, not a hand-edit. +2. **Error-message version — resolved.** `error/codes` is now Proxy-wrapped so each code's `message` + resolves lazily from the identity-aware `errorMessages` at throw time. Each layer stamps its + identity from an in-source `version.ts` constant (kept in sync by the bump script), so error text + self-reports the live version: skyvault → `v2.1.2`, flowvault → `v1.0.0`. Note the error-prefix + *name* ("Skyflow Node SDK") is still a shared string (keeping skyvault backward-compatible with + its `removeSDKVersion` regex); the per-product *name* is reported in telemetry (`sdk_name_version`). +3. **`src/ _generated_` leading space** — the legacy skyvault data-plane generated directory name + contains a leading space; it is preserved (not renamed) and remains wired into jest/eslint/ + cspell/gitleaks patterns. + +### FlowVault contract — reconciled against the Java & Python flowvault SDKs + +The flowvault layer was first written to the frozen design doc, then **aligned to the Java and +Python flowvault SDKs** (the authoritative cross-language contract). Final decisions: + +| Topic | Decision (aligned to Java/Python) | +| --- | --- | +| **Errors** | Throw a flat camelCase `SkyflowError` (`httpCode` from HTTP status, `requestId` from the `x-request-id` header, `grpcCode`/`httpStatus`/`message`/`details`). The wire body is snake_case under `error`; the SDK reads it and surfaces camelCase. No nested `error` key on the thrown object. Matches Java `SkyflowException`. | +| **BYOT `tokens` (insert/update)** | **Kept** — Java flowvault's `InsertRequestRecord`/`UpdateRequestRecord` carry `tokens`. The Node generated `InsertRecordData`/`UpdateRecordData` now model `tokens?` (regenerated), so the SDK forwards it typed (no cast). ✅ resolved. | +| **Detokenize response** | Read wire key `response[]`, expose as `records[]`. Matches Java/Python. | +| **Delete response** | **No `tableName`** on `DeleteResponseRecord` (only `skyflowId`/`httpCode`/`error`/`requestId`). Matches Java `DeleteResponseRecord` and Python `DeleteResponseRecord`. | +| **`uniqueValues` (get & delete)** | A **list of `{ column: value, … }` maps**, each sent as `{ data: {…} }`. (Not `{columnName, columnValues[]}`.) Matches Java `List` → `V1UniqueValue{data}` and Python `__to_unique_values`. | +| **deleteFiles column status** | **Parsed** from each column's response value (object → `.status`, else the value), not assumed `"DELETED"`. Mirrors Python `__file_column_status`. (`DeleteFilesResponseRecord` keeps `tableName` + `columns` — unlike record-delete.) | +| **insert/update `fields`** | Generated `RecordResponseObject` splits `data` + `tokens`; the SDK exposes one `fields = tokens ?? data`. (Retained merge choice.) | +| **Query** | flowvault **has** `query` (as in Python; Java flowvault has none). Response is `fields`/`records` + `metadata.columns` + call-level `requestId`; **no per-record `errors`** (the query API has no partial-error responses, so `errors` is always `null`). | + +Open regeneration items (generated code untouched; flag for the maintainer): +- Both `common` and `flowvault` generated clients ship a top-level `Client.ts` (`patchV2VaultsId`) + that fails `strict` (`encodeURIComponent(vaultId)` on `string | undefined`); unused by the + hand-written layers (which compose resource clients directly), so it + its `index.ts` are + `exclude`d in the relevant tsconfigs — a regeneration concern, not a hand-edit. + +🤖 Generated with [Claude Code](https://claude.com/claude-code) diff --git a/eslint.config.mjs b/eslint.config.mjs index 7d4fde46..b91f395e 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -41,13 +41,17 @@ export default tseslint.config( , { ignores: [ - "node_modules/", - "dist/", - "coverage/", - "lib/", - "src/ _generated_/", - "test/**", - "samples/**", + "**/node_modules/**", + "**/dist/**", + "**/coverage/**", + "**/lib/**", + // Generated (maintainer-owned) clients in every workspace — both the + // plain and the legacy leading-space directory names. + "**/_generated_/**", + "**/ _generated_/**", + // Tests live under each workspace (common/test, skyvault/test, flowvault/test). + "**/test/**", + "**/samples/**", "scripts/**", ] } diff --git a/flowvault/.gitignore b/flowvault/.gitignore new file mode 100644 index 00000000..80e178c4 --- /dev/null +++ b/flowvault/.gitignore @@ -0,0 +1,3 @@ + +# npm pack artifacts (local) +*.tgz diff --git a/flowvault/LICENSE b/flowvault/LICENSE new file mode 100644 index 00000000..189f34b5 --- /dev/null +++ b/flowvault/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2021 Skyflow Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/flowvault/README.md b/flowvault/README.md new file mode 100644 index 00000000..323a9adf --- /dev/null +++ b/flowvault/README.md @@ -0,0 +1,348 @@ +# Skyflow FlowVault Node SDK + +The FlowVault (v2) Node.js SDK for Skyflow's data privacy vault — a unary, strongly-typed client +for the v2 vault data plane. With it you can securely insert, retrieve, update, delete, query, +detokenize, tokenize, and attach files to records in a Flow DB vault. + +> This is a **separate product** from `skyflow-node` (the SkyVault SDK). Install exactly one: +> `npm i skyflow-flowvault-node` **or** `npm i skyflow-node`. A lower `flowvault` version number +> does not mean it is older — the two products version independently. + +## Table of contents + +- [Overview](#overview) +- [Install](#install) +- [Quickstart](#quickstart) +- [Authenticate](#authenticate) +- [Initialize the client](#initialize-the-client) +- [Vault operations](#vault-operations) +- [Custom request headers](#custom-request-headers) +- [Error handling](#error-handling) +- [Samples](#samples) +- [License](#license) + +## Overview + +- **Unary interfaces only.** Every method is a single request → single response; there are no bulk + or async variants. +- **Unified responses.** Each response carries a single `records` list, with per-record `httpCode`, + `error`, and `requestId`. Whole-call failures throw a `SkyflowError`. +- **Multi-vault.** One client can hold several vault configs; pick one per call with `vault(vaultId)`. + +## Install + +```bash +npm install skyflow-flowvault-node +``` + +**Requirements:** Node.js 20+ (the `File` API used by `uploadFiles` requires v20+). + +## Quickstart + +```typescript +import { Skyflow, InsertRequest, Env, LogLevel, SkyflowError } from 'skyflow-flowvault-node'; + +const skyflowClient = new Skyflow({ + vaultConfigs: [{ + vaultId: 'your-vault-id', + clusterId: 'your-cluster-id', // from the vault URL: https://{clusterId}.vault.skyflowapis.com + env: Env.PROD, // DEV, STAGE, SANDBOX, or PROD + credentials: { token: 'your-bearer-token' }, + }], + logLevel: LogLevel.ERROR, +}); + +try { + const req = new InsertRequest('cards', [ + { data: { card_number: '4111111111111111', cardholder_name: 'Jane Doe' } }, + ]); + const res = await skyflowClient.vault('your-vault-id').insert(req); + console.log(res.records); +} catch (error) { + if (error instanceof SkyflowError) console.error(error.error?.httpCode, error.message); +} +``` + +## Authenticate + +### Credential types + +Pass exactly one credential type in a vault config's `credentials`: + +```typescript +// API key +const credentials = { apiKey: 'your-api-key' }; + +// Bearer token you generated yourself +const credentials = { token: 'your-bearer-token' }; + +// Service-account credentials file — the SDK generates and refreshes the token +const credentials = { path: '/path/to/credentials.json' }; + +// Service-account credentials as a JSON string +const credentials = { credentialsString: JSON.stringify(serviceAccountCreds) }; +``` + +If no `credentials` are set on a vault config, the SDK reads the `SKYFLOW_CREDENTIALS` environment +variable. + +### Generate tokens yourself + +The service-account helpers are re-exported from this package: + +```typescript +import { + generateBearerToken, + generateBearerTokenFromCreds, + generateSignedDataTokens, + isExpired, +} from 'skyflow-flowvault-node'; + +// From a credentials file path (reuse until it expires): +if (isExpired(bearerToken)) { + const { accessToken } = await generateBearerToken('/path/to/credentials.json'); + bearerToken = accessToken; +} + +// Scoped token (restrict to roles): +const { accessToken } = await generateBearerToken('/path/to/credentials.json', { + roleIds: ['roleId1', 'roleId2'], +}); +``` + +See [`samples/service-account/`](samples/service-account) for scoped and signed-token examples. + +## Initialize the client + +```typescript +import { Skyflow, Env, LogLevel } from 'skyflow-flowvault-node'; + +const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: 'vault-id', // required + clusterId: 'cluster-id', // required — from the vault URL + env: Env.PROD, // optional — DEV | STAGE | SANDBOX | PROD (default PROD) + credentials: { token: 'bearer-token' }, // optional — falls back to SKYFLOW_CREDENTIALS + }, + ], + logLevel: LogLevel.ERROR, // optional — DEBUG | INFO | WARN | ERROR (default) | OFF +}); + +const vault = skyflowClient.vault('vault-id'); // omit the id to use the first configured vault +``` + +## Vault operations + +All nine operations are unary and return a single `records` list (query returns `fields`). The +snippets below are abbreviated — see [`samples/vault-api/`](samples/vault-api) for complete, +runnable files. + +### Insert + +```typescript +import { InsertRequest, InsertOptions } from 'skyflow-flowvault-node'; + +const req = new InsertRequest('cards', [ + { data: { card_number: '4111111111111111' } }, + // Per-record table + upsert — insert across tables in one call. When any record + // sets its own upsert, the request-level upsert below is ignored: + // { data: { email: 'jane@example.com' }, table: 'contacts', + // upsert: { updateType: 'UPDATE', uniqueColumns: ['email'] } }, +]); + +// Request-level upsert (applies to all records unless a record sets its own): +const options = new InsertOptions(); +// options.setUpsert({ updateType: 'UPDATE', uniqueColumns: ['email'] }); + +const res = await vault.insert(req, options); +// res.records[i]: { tableName, skyflowId, tokens, data, hashedData, httpCode, error, requestId } +``` + +### Get + +```typescript +import { GetRequest, GetOptions } from 'skyflow-flowvault-node'; + +// single-table — table + ids, with optional columns / redactions / unique values / limit / offset +const req = new GetRequest('cards', ['skyflow-id-1', 'skyflow-id-2']); +const options = new GetOptions(); +// options.setFields(['card_number']); +// options.setColumnRedactions([{ columnName: 'card_number', redaction: 'MASKED' }]); +// options.setUniqueValues([{ email: 'jane@example.com' }]); +const res = await vault.get(req, options); + +// multi-table batch — each record carries its own table + lookup/column fields: +// const res = await vault.get(new GetRequest('', [], [ +// { tableName: 'cards', ids: ['skyflow-id-1'], columns: ['card_number'] }, +// { tableName: 'contacts', ids: [], uniqueValues: [{ email: 'jane@example.com' }] }, +// ])); +``` + +### Update + +```typescript +import { UpdateRequest } from 'skyflow-flowvault-node'; + +// updateType (optional, request-level): 'UPDATE' (merge columns) or 'REPLACE' (overwrite record). +const req = new UpdateRequest('cards', [ + { skyflowId: 'skyflow-id', data: { cardholder_name: 'Jane Doe' } }, +], 'UPDATE'); +const res = await vault.update(req); +``` + +### Delete + +```typescript +import { DeleteRequest } from 'skyflow-flowvault-node'; + +const req = new DeleteRequest('cards', ['skyflow-id-1']); +// Or by unique values: new DeleteRequest('cards', [], [{ email: 'john@example.com' }]); +const res = await vault.delete(req); +``` + +### Query + +```typescript +import { QueryRequest } from 'skyflow-flowvault-node'; + +// SELECT only; at most 25 rows per call — page with OFFSET. +const req = new QueryRequest("SELECT name, email FROM cards LIMIT 25"); +const res = await vault.query(req); +// res.fields (rows), res.metadata.columns, res.requestId +``` + +### Detokenize + +```typescript +import { DetokenizeRequest } from 'skyflow-flowvault-node'; + +const req = new DetokenizeRequest(['token-1', 'token-2']); +// Optional group redactions: new DetokenizeRequest(tokens, [{ tokenGroupName: 'g', redaction: 'PLAIN_TEXT' }]); +const res = await vault.detokenize(req); +``` + +### Get tokens + +```typescript +import { GetTokensRequest } from 'skyflow-flowvault-node'; + +// Deterministic token groups only. +const req = new GetTokensRequest([{ value: 'john@example.com', tokenGroupName: 'group' }]); +const res = await vault.getTokens(req); +``` + +### Upload files + +```typescript +import { UploadFilesRequest } from 'skyflow-flowvault-node'; + +// Each column takes exactly ONE of filePath / base64 (needs fileName) / file. +const req = new UploadFilesRequest([ + { tableName: 'onboarding', skyflowId: 'skyflow-id', columns: [{ column: 'resume', filePath: '/tmp/resume.pdf' }] }, +]); +const res = await vault.uploadFiles(req); +// each column result: { column, fileName, uploadStatus: 'UPLOADED'|'FAILED'|'SKIPPED', error } +``` + +### Delete files + +```typescript +import { DeleteFilesRequest } from 'skyflow-flowvault-node'; + +// Each record sets exactly ONE of skyflowId or uniqueValues. +const req = new DeleteFilesRequest([ + { tableName: 'onboarding', columns: ['resume'], skyflowId: 'skyflow-id' }, +]); +const res = await vault.deleteFiles(req); +``` + +## Custom request headers + +The SDK supports a fixed set of custom headers, listed on the `CustomHeaderKey` +enum. Attach them per call by registering an interceptor on the `*Options` +object — it runs once per request and receives a `RequestContext`: + +| `CustomHeaderKey` | Header sent | +| ---------------------- | ---------------------- | +| `SKYFLOW_ACCOUNT_ID` | `x-skyflow-account-id` | +| `SKYFLOW_ACCOUNT_NAME` | `x-skyflow-account-name` | +| `REQUEST_ID_HEADER` | `x-request-id` | + +```typescript +import { InsertOptions, CustomHeaderKey } from 'skyflow-flowvault-node'; +import { randomUUID } from 'crypto'; + +const options = new InsertOptions(); +options.setInterceptor((ctx) => { + // ctx.operation names the call ('INSERT', 'GET', ...), so headers can vary per operation. + ctx.addHeader(CustomHeaderKey.REQUEST_ID_HEADER, randomUUID()); +}); + +await vault.insert(req, options); +``` + +Only the keys above are accepted; arbitrary header names are not supported. + +## Timeouts and retries + +Configure per-vault timeout and retry behaviour on the vault config: + +```typescript +const skyflowClient = new Skyflow({ + vaultConfigs: [{ + vaultId: 'vault-id', + clusterId: 'cluster-id', + credentials: { token: 'bearer-token' }, + timeout: 30, // overall per-call timeout in seconds (incl. retries). Default 60. + maxRetries: 2, // retry attempts after the first failure (408/429/5xx). Default off. + vaultUrl: 'https://cluster-id.skyvault.skyflowapis.com', // optional — override the derived URL + }], +}); +``` + +Retries use exponential backoff with jitter. The connect/read/write timeouts and the backoff +delays are fixed in the SDK's request layer and are not individually configurable. + +## Error handling + +There are two layers: + +- **Per-record** — an individual record can fail while the call succeeds. Each response record has + `httpCode` and `error` (plus `requestId` when it failed). Inspect them per record. +- **Whole-call** — auth, networking, or request-shape failures throw a `SkyflowError`. + +```typescript +import { SkyflowError } from 'skyflow-flowvault-node'; + +try { + const res = await vault.insert(req); + for (const record of res.records) { + if (record.error != null) { + console.error(`record failed (${record.httpCode}): ${record.error} [${record.requestId}]`); + } + } +} catch (error) { + if (error instanceof SkyflowError) { + console.error({ + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected error:', error); + } +} +``` + +## Samples + +Runnable, one-file-per-operation examples live in [`samples/`](samples) — see +[`samples/README.md`](samples/README.md). + +## License + +This SDK is licensed under the MIT License. diff --git a/flowvault/_generated_/rest/src/Client.ts b/flowvault/_generated_/rest/src/Client.ts new file mode 100644 index 00000000..0bd4bec0 --- /dev/null +++ b/flowvault/_generated_/rest/src/Client.ts @@ -0,0 +1,142 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "./environments"; +import * as core from "./core"; +import * as Skyflow from "./api/index"; +import urlJoin from "url-join"; +import * as errors from "./errors/index"; +import { Files } from "./api/resources/files/client/Client"; +import { Query } from "./api/resources/query/client/Client"; +import { Records } from "./api/resources/records/client/Client"; +import { Tokens } from "./api/resources/tokens/client/Client"; + +export declare namespace SkyflowClient { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class SkyflowClient { + protected _files: Files | undefined; + protected _query: Query | undefined; + protected _records: Records | undefined; + protected _tokens: Tokens | undefined; + + constructor(protected readonly _options: SkyflowClient.Options = {}) {} + + public get files(): Files { + return (this._files ??= new Files(this._options)); + } + + public get query(): Query { + return (this._query ??= new Query(this._options)); + } + + public get records(): Records { + return (this._records ??= new Records(this._options)); + } + + public get tokens(): Tokens { + return (this._tokens ??= new Tokens(this._options)); + } + + /** + * @param {string | undefined} vaultId + * @param {Skyflow.PatchV2VaultsIdRequest} request + * @param {SkyflowClient.RequestOptions} requestOptions - Request-specific configuration. + * + * @example + * await client.patchV2VaultsId(undefined) + */ + public patchV2VaultsId( + vaultId: string | undefined, + request: Skyflow.PatchV2VaultsIdRequest = {}, + requestOptions?: SkyflowClient.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__patchV2VaultsId(vaultId, request, requestOptions)); + } + + private async __patchV2VaultsId( + vaultId: string | undefined, + request: Skyflow.PatchV2VaultsIdRequest = {}, + requestOptions?: SkyflowClient.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + `v2/vaults/${encodeURIComponent(vaultId)}`, + ), + method: "PATCH", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: undefined, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling PATCH /v2/vaults/{vaultID}."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/src/ _generated_/rest/api/resources/binLookup/client/index.ts b/flowvault/_generated_/rest/src/api/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/binLookup/client/index.ts rename to flowvault/_generated_/rest/src/api/client/index.ts diff --git a/flowvault/_generated_/rest/src/api/client/requests/PatchV2VaultsIdRequest.ts b/flowvault/_generated_/rest/src/api/client/requests/PatchV2VaultsIdRequest.ts new file mode 100644 index 00000000..a2929a18 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/client/requests/PatchV2VaultsIdRequest.ts @@ -0,0 +1,9 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * {} + */ +export interface PatchV2VaultsIdRequest {} diff --git a/flowvault/_generated_/rest/src/api/client/requests/index.ts b/flowvault/_generated_/rest/src/api/client/requests/index.ts new file mode 100644 index 00000000..1a0e15b2 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/client/requests/index.ts @@ -0,0 +1 @@ +export { type PatchV2VaultsIdRequest } from "./PatchV2VaultsIdRequest"; diff --git a/flowvault/_generated_/rest/src/api/errors/BadRequestError.ts b/flowvault/_generated_/rest/src/api/errors/BadRequestError.ts new file mode 100644 index 00000000..1f9b3de6 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/errors/BadRequestError.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; +import * as core from "../../core"; + +export class BadRequestError extends errors.SkyflowError { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { + super({ + message: "BadRequestError", + statusCode: 400, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, BadRequestError.prototype); + } +} diff --git a/flowvault/_generated_/rest/src/api/errors/ForbiddenError.ts b/flowvault/_generated_/rest/src/api/errors/ForbiddenError.ts new file mode 100644 index 00000000..699b312d --- /dev/null +++ b/flowvault/_generated_/rest/src/api/errors/ForbiddenError.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; +import * as core from "../../core"; + +export class ForbiddenError extends errors.SkyflowError { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { + super({ + message: "ForbiddenError", + statusCode: 403, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, ForbiddenError.prototype); + } +} diff --git a/src/ _generated_/rest/api/errors/InternalServerError.ts b/flowvault/_generated_/rest/src/api/errors/InternalServerError.ts similarity index 100% rename from src/ _generated_/rest/api/errors/InternalServerError.ts rename to flowvault/_generated_/rest/src/api/errors/InternalServerError.ts diff --git a/flowvault/_generated_/rest/src/api/errors/NotFoundError.ts b/flowvault/_generated_/rest/src/api/errors/NotFoundError.ts new file mode 100644 index 00000000..f5226d35 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/errors/NotFoundError.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; +import * as core from "../../core"; + +export class NotFoundError extends errors.SkyflowError { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { + super({ + message: "NotFoundError", + statusCode: 404, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, NotFoundError.prototype); + } +} diff --git a/flowvault/_generated_/rest/src/api/errors/TooManyRequestsError.ts b/flowvault/_generated_/rest/src/api/errors/TooManyRequestsError.ts new file mode 100644 index 00000000..879bed1c --- /dev/null +++ b/flowvault/_generated_/rest/src/api/errors/TooManyRequestsError.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; +import * as core from "../../core"; + +export class TooManyRequestsError extends errors.SkyflowError { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { + super({ + message: "TooManyRequestsError", + statusCode: 429, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, TooManyRequestsError.prototype); + } +} diff --git a/flowvault/_generated_/rest/src/api/errors/UnauthorizedError.ts b/flowvault/_generated_/rest/src/api/errors/UnauthorizedError.ts new file mode 100644 index 00000000..9d763249 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/errors/UnauthorizedError.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; +import * as core from "../../core"; + +export class UnauthorizedError extends errors.SkyflowError { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { + super({ + message: "UnauthorizedError", + statusCode: 401, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, UnauthorizedError.prototype); + } +} diff --git a/flowvault/_generated_/rest/src/api/errors/index.ts b/flowvault/_generated_/rest/src/api/errors/index.ts new file mode 100644 index 00000000..9dc2c3b7 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/errors/index.ts @@ -0,0 +1,6 @@ +export * from "./BadRequestError"; +export * from "./UnauthorizedError"; +export * from "./ForbiddenError"; +export * from "./NotFoundError"; +export * from "./TooManyRequestsError"; +export * from "./InternalServerError"; diff --git a/flowvault/_generated_/rest/src/api/index.ts b/flowvault/_generated_/rest/src/api/index.ts new file mode 100644 index 00000000..d5c609c4 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/index.ts @@ -0,0 +1,4 @@ +export * from "./resources"; +export * from "./types"; +export * from "./errors"; +export * from "./client"; diff --git a/flowvault/_generated_/rest/src/api/resources/files/client/Client.ts b/flowvault/_generated_/rest/src/api/resources/files/client/Client.ts new file mode 100644 index 00000000..f38567a7 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/files/client/Client.ts @@ -0,0 +1,290 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace Files { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class Files { + constructor(protected readonly _options: Files.Options = {}) {} + + /** + * Deletes one or more files from the specified file column(s) of existing record(s) in a vault, and removes the underlying object from cloud storage. + * + * @param {Skyflow.FileDeleteRequest} request + * @param {Files.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.ForbiddenError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.TooManyRequestsError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.files.deleteFiles({ + * vaultID: "d408485953784308a000f8dcf81901ef", + * records: [{ + * tableName: "onboarding", + * skyflowID: "77ea0577-894a-4a42-90a9-51b0eaf97159", + * columns: ["resumePDF", "photoID"] + * }] + * }) + */ + public deleteFiles( + request: Skyflow.FileDeleteRequest, + requestOptions?: Files.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__deleteFiles(request, requestOptions)); + } + + private async __deleteFiles( + request: Skyflow.FileDeleteRequest, + requestOptions?: Files.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/files/delete", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.FileDeleteResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 403: + throw new Skyflow.ForbiddenError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 429: + throw new Skyflow.TooManyRequestsError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/files/delete."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Generates signed URLs for uploading files to specified columns in a vault. Returns a signed URL per column that expires in 15 minutes by default. If a skyflowID is not provided for a record, a new record is created and its skyflowID is returned. + * + * @param {Skyflow.FileUploadRequest} request + * @param {Files.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.ForbiddenError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.TooManyRequestsError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.files.uploadFiles({ + * vaultID: "d408485953784308a000f8dcf81901ef", + * records: [{ + * tableName: "onboarding", + * columns: [{ + * column: "resumePDF", + * fileName: "sampleresume.pdf" + * }, { + * column: "photoID", + * fileName: "samplephoto.jpg" + * }] + * }] + * }) + */ + public uploadFiles( + request: Skyflow.FileUploadRequest, + requestOptions?: Files.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__uploadFiles(request, requestOptions)); + } + + private async __uploadFiles( + request: Skyflow.FileUploadRequest, + requestOptions?: Files.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/files/upload", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.FileUploadResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 403: + throw new Skyflow.ForbiddenError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 429: + throw new Skyflow.TooManyRequestsError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/files/upload."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/src/ _generated_/rest/api/resources/files/client/index.ts b/flowvault/_generated_/rest/src/api/resources/files/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/index.ts rename to flowvault/_generated_/rest/src/api/resources/files/client/index.ts diff --git a/flowvault/_generated_/rest/src/api/resources/files/client/requests/FileDeleteRequest.ts b/flowvault/_generated_/rest/src/api/resources/files/client/requests/FileDeleteRequest.ts new file mode 100644 index 00000000..4472e273 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/files/client/requests/FileDeleteRequest.ts @@ -0,0 +1,23 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef", + * records: [{ + * tableName: "onboarding", + * skyflowID: "77ea0577-894a-4a42-90a9-51b0eaf97159", + * columns: ["resumePDF", "photoID"] + * }] + * } + */ +export interface FileDeleteRequest { + /** ID of the vault. */ + vaultID: string; + /** List of record objects (max 20,000) identifying the file columns to delete. */ + records: Skyflow.FileDeleteRecord[]; +} diff --git a/flowvault/_generated_/rest/src/api/resources/files/client/requests/FileUploadRequest.ts b/flowvault/_generated_/rest/src/api/resources/files/client/requests/FileUploadRequest.ts new file mode 100644 index 00000000..8f8235ba --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/files/client/requests/FileUploadRequest.ts @@ -0,0 +1,28 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef", + * records: [{ + * tableName: "onboarding", + * columns: [{ + * column: "resumePDF", + * fileName: "sampleresume.pdf" + * }, { + * column: "photoID", + * fileName: "samplephoto.jpg" + * }] + * }] + * } + */ +export interface FileUploadRequest { + /** ID of the vault. */ + vaultID: string; + /** List of records for which signed file upload URLs are requested. */ + records: Skyflow.FileUploadRecord[]; +} diff --git a/flowvault/_generated_/rest/src/api/resources/files/client/requests/index.ts b/flowvault/_generated_/rest/src/api/resources/files/client/requests/index.ts new file mode 100644 index 00000000..c073ac4e --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/files/client/requests/index.ts @@ -0,0 +1,2 @@ +export { type FileDeleteRequest } from "./FileDeleteRequest"; +export { type FileUploadRequest } from "./FileUploadRequest"; diff --git a/src/ _generated_/rest/api/resources/binLookup/index.ts b/flowvault/_generated_/rest/src/api/resources/files/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/binLookup/index.ts rename to flowvault/_generated_/rest/src/api/resources/files/index.ts diff --git a/flowvault/_generated_/rest/src/api/resources/index.ts b/flowvault/_generated_/rest/src/api/resources/index.ts new file mode 100644 index 00000000..924c424a --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/index.ts @@ -0,0 +1,9 @@ +export * as records from "./records"; +export * from "./records/types"; +export * as files from "./files"; +export * as query from "./query"; +export * as tokens from "./tokens"; +export * from "./files/client/requests"; +export * from "./query/client/requests"; +export * from "./records/client/requests"; +export * from "./tokens/client/requests"; diff --git a/flowvault/_generated_/rest/src/api/resources/query/client/Client.ts b/flowvault/_generated_/rest/src/api/resources/query/client/Client.ts new file mode 100644 index 00000000..a8fdb633 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/query/client/Client.ts @@ -0,0 +1,154 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace Query { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class Query { + constructor(protected readonly _options: Query.Options = {}) {} + + /** + * Returns records for a valid SQL query. This endpoint + * - Can return masked record values. + * - Supports only the `SELECT` command. + * - Returns a maximum of 25 records. To return additional records, perform another query using the `OFFSET` keyword. + * - Can't modify the vault or perform transactions. + * - Can't return tokens. + * - Can't return file download or render URLs. + * + * @param {Skyflow.ExecuteQueryRequest} request + * @param {Query.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.query.executeQuery({ + * vaultID: "d408485953784308a000f8dcf81901ef", + * query: "query" + * }) + */ + public executeQuery( + request: Skyflow.ExecuteQueryRequest, + requestOptions?: Query.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__executeQuery(request, requestOptions)); + } + + private async __executeQuery( + request: Skyflow.ExecuteQueryRequest, + requestOptions?: Query.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/query", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.ExecuteQueryResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/query."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/src/ _generated_/rest/api/resources/guardrails/client/index.ts b/flowvault/_generated_/rest/src/api/resources/query/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/guardrails/client/index.ts rename to flowvault/_generated_/rest/src/api/resources/query/client/index.ts diff --git a/flowvault/_generated_/rest/src/api/resources/query/client/requests/ExecuteQueryRequest.ts b/flowvault/_generated_/rest/src/api/resources/query/client/requests/ExecuteQueryRequest.ts new file mode 100644 index 00000000..08494057 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/query/client/requests/ExecuteQueryRequest.ts @@ -0,0 +1,17 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef", + * query: "query" + * } + */ +export interface ExecuteQueryRequest { + /** ID of the vault where the query is being performed. */ + vaultID: string; + /** Query to perform. */ + query: string; +} diff --git a/flowvault/_generated_/rest/src/api/resources/query/client/requests/index.ts b/flowvault/_generated_/rest/src/api/resources/query/client/requests/index.ts new file mode 100644 index 00000000..22e2aaab --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/query/client/requests/index.ts @@ -0,0 +1 @@ +export { type ExecuteQueryRequest } from "./ExecuteQueryRequest"; diff --git a/src/ _generated_/rest/api/resources/guardrails/index.ts b/flowvault/_generated_/rest/src/api/resources/query/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/guardrails/index.ts rename to flowvault/_generated_/rest/src/api/resources/query/index.ts diff --git a/flowvault/_generated_/rest/src/api/resources/records/client/Client.ts b/flowvault/_generated_/rest/src/api/resources/records/client/Client.ts new file mode 100644 index 00000000..a717b2ef --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/client/Client.ts @@ -0,0 +1,541 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace Records { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class Records { + constructor(protected readonly _options: Records.Options = {}) {} + + /** + * Deletes records from a vault. + * + * @param {Skyflow.DeleteRequest} request + * @param {Records.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.records.deleteRecords({ + * vaultID: "d408485953784308a000f8dcf81901ef" + * }) + */ + public deleteRecords( + request: Skyflow.DeleteRequest, + requestOptions?: Records.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__deleteRecords(request, requestOptions)); + } + + private async __deleteRecords( + request: Skyflow.DeleteRequest, + requestOptions?: Records.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/records/delete", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.DeleteResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/records/delete."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Returns the specified records from a vault. + * + * @param {Skyflow.GetRequest} request + * @param {Records.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.records.getRecords({ + * vaultID: "d408485953784308a000f8dcf81901ef" + * }) + */ + public getRecords( + request: Skyflow.GetRequest, + requestOptions?: Records.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__getRecords(request, requestOptions)); + } + + private async __getRecords( + request: Skyflow.GetRequest, + requestOptions?: Records.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/records/get", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.GetResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/records/get."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Inserts new records into a vault. + * + * @param {Skyflow.InsertRequest} request + * @param {Records.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.records.insertRecords({ + * vaultID: "d408485953784308a000f8dcf81901ef", + * tableName: "employees", + * records: [{ + * data: { + * "name": "Vivek1", + * "email": "vivek.varshney@skyflow.com", + * "age": 23, + * "adult": true, + * "address": { + * "street": "Bata Gali", + * "city": "Aligarh", + * "state": "UP", + * "postal_code": 202001, + * "country": "India", + * "phone_numbers": [ + * { + * "type": "home", + * "number": [ + * 1234, + * 5678 + * ] + * }, + * { + * "type": "work", + * "number": [ + * 4321, + * 8765 + * ] + * } + * ] + * } + * } + * }, { + * data: { + * "name": "Asad1", + * "email": "asad.public@gmail.com", + * "age": 16, + * "adult": false, + * "address": { + * "street": "Sarojini", + * "city": "Deoria", + * "state": "UP", + * "postal_code": 274001, + * "country": "India", + * "phone_numbers": [ + * { + * "type": "home", + * "number": [ + * 7890, + * 3456 + * ] + * }, + * { + * "type": "work", + * "number": [ + * 9087, + * 6543 + * ] + * } + * ] + * } + * } + * }] + * }) + */ + public insertRecords( + request: Skyflow.InsertRequest, + requestOptions?: Records.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__insertRecords(request, requestOptions)); + } + + private async __insertRecords( + request: Skyflow.InsertRequest, + requestOptions?: Records.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/records/insert", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.InsertResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/records/insert."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Updates the specified records in a vault. + * + * @param {Skyflow.UpdateRequest} request + * @param {Records.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.records.updateRecords({ + * vaultID: "d408485953784308a000f8dcf81901ef", + * tableName: "employees", + * records: [{ + * skyflowID: "97cdd1af-02ac-47eb-ab0d-8339dbef6ccb", + * data: { + * "name": "Vivek", + * "email": "asad.public5@gmail.com", + * "age": 25, + * "adult": true, + * "address": { + * "city": "aligarh", + * "country": "India" + * } + * } + * }, { + * skyflowID: "aed32bbc-e7a4-4c7b-8c88-7f52ce8cb066", + * data: { + * "name": "Asad", + * "email": "asad.public7@gmail.com", + * "age": 24, + * "adult": false, + * "address": { + * "city": "deoria", + * "country": "India" + * } + * } + * }] + * }) + */ + public updateRecords( + request: Skyflow.UpdateRequest, + requestOptions?: Records.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__updateRecords(request, requestOptions)); + } + + private async __updateRecords( + request: Skyflow.UpdateRequest, + requestOptions?: Records.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/records/update", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.UpdateResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/records/update."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/src/ _generated_/rest/api/resources/query/client/index.ts b/flowvault/_generated_/rest/src/api/resources/records/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/query/client/index.ts rename to flowvault/_generated_/rest/src/api/resources/records/client/index.ts diff --git a/flowvault/_generated_/rest/src/api/resources/records/client/requests/DeleteRequest.ts b/flowvault/_generated_/rest/src/api/resources/records/client/requests/DeleteRequest.ts new file mode 100644 index 00000000..6c528dbb --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/client/requests/DeleteRequest.ts @@ -0,0 +1,22 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef" + * } + */ +export interface DeleteRequest { + /** ID of the vault. */ + vaultID: string; + /** Name of the table. */ + tableName?: string; + /** Skyflow IDs of the records to delete. */ + skyflowIDs?: string[]; + /** List of unique constraint values to query records by data. */ + uniqueValues?: Skyflow.UniqueValue[]; +} diff --git a/flowvault/_generated_/rest/src/api/resources/records/client/requests/GetRequest.ts b/flowvault/_generated_/rest/src/api/resources/records/client/requests/GetRequest.ts new file mode 100644 index 00000000..29cb25a5 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/client/requests/GetRequest.ts @@ -0,0 +1,32 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef" + * } + */ +export interface GetRequest { + /** ID of the vault. */ + vaultID: string; + /** Name of the table to perform the operation on. */ + tableName?: string; + /** Skyflow IDs of the records to return. Either `skyflowIDs` or `uniqueValues` are required. If both are provided, the request fails. */ + skyflowIDs?: string[]; + /** List of columns to redact. */ + columnRedactions?: Skyflow.ColumnRedactions[]; + /** List of columns to return. */ + columns?: string[]; + /** Limit for the number of records to be fetched. */ + limit?: number; + /** Offset for the number of records to be fetched. */ + offset?: number; + /** List of unique constraint values to query records by data. */ + uniqueValues?: Skyflow.UniqueValue[]; + /** List of records to be fetched. This field contains tableName and skyflowIDs belonging to the table. */ + records?: Skyflow.GetRequestData[]; +} diff --git a/flowvault/_generated_/rest/src/api/resources/records/client/requests/InsertRequest.ts b/flowvault/_generated_/rest/src/api/resources/records/client/requests/InsertRequest.ts new file mode 100644 index 00000000..e89511dc --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/client/requests/InsertRequest.ts @@ -0,0 +1,83 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef", + * tableName: "employees", + * records: [{ + * data: { + * "name": "Vivek1", + * "email": "vivek.varshney@skyflow.com", + * "age": 23, + * "adult": true, + * "address": { + * "street": "Bata Gali", + * "city": "Aligarh", + * "state": "UP", + * "postal_code": 202001, + * "country": "India", + * "phone_numbers": [ + * { + * "type": "home", + * "number": [ + * 1234, + * 5678 + * ] + * }, + * { + * "type": "work", + * "number": [ + * 4321, + * 8765 + * ] + * } + * ] + * } + * } + * }, { + * data: { + * "name": "Asad1", + * "email": "asad.public@gmail.com", + * "age": 16, + * "adult": false, + * "address": { + * "street": "Sarojini", + * "city": "Deoria", + * "state": "UP", + * "postal_code": 274001, + * "country": "India", + * "phone_numbers": [ + * { + * "type": "home", + * "number": [ + * 7890, + * 3456 + * ] + * }, + * { + * "type": "work", + * "number": [ + * 9087, + * 6543 + * ] + * } + * ] + * } + * } + * }] + * } + */ +export interface InsertRequest { + /** ID of the vault. */ + vaultID: string; + /** Name of the table to perform the operation on. Can be defined at both the request body level and individual record level. If provided at both levels, the record-level `tableName` takes precedence. */ + tableName: string; + /** Data to insert as a list of records. */ + records: Skyflow.InsertRecordData[]; + upsert?: Skyflow.Upsert; +} diff --git a/flowvault/_generated_/rest/src/api/resources/records/client/requests/UpdateRequest.ts b/flowvault/_generated_/rest/src/api/resources/records/client/requests/UpdateRequest.ts new file mode 100644 index 00000000..a952479f --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/client/requests/UpdateRequest.ts @@ -0,0 +1,48 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef", + * tableName: "employees", + * records: [{ + * skyflowID: "97cdd1af-02ac-47eb-ab0d-8339dbef6ccb", + * data: { + * "name": "Vivek", + * "email": "asad.public5@gmail.com", + * "age": 25, + * "adult": true, + * "address": { + * "city": "aligarh", + * "country": "India" + * } + * } + * }, { + * skyflowID: "aed32bbc-e7a4-4c7b-8c88-7f52ce8cb066", + * data: { + * "name": "Asad", + * "email": "asad.public7@gmail.com", + * "age": 24, + * "adult": false, + * "address": { + * "city": "deoria", + * "country": "India" + * } + * } + * }] + * } + */ +export interface UpdateRequest { + /** ID of the vault. */ + vaultID: string; + /** Name of the table. */ + tableName: string; + /** Data to update as a list of records. */ + records: Skyflow.UpdateRecordData[]; + /** Type of update operation to perform. */ + updateType?: Skyflow.UpdateRequestUpdateType; +} diff --git a/flowvault/_generated_/rest/src/api/resources/records/client/requests/index.ts b/flowvault/_generated_/rest/src/api/resources/records/client/requests/index.ts new file mode 100644 index 00000000..29bbd957 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/client/requests/index.ts @@ -0,0 +1,4 @@ +export { type DeleteRequest } from "./DeleteRequest"; +export { type GetRequest } from "./GetRequest"; +export { type InsertRequest } from "./InsertRequest"; +export { type UpdateRequest } from "./UpdateRequest"; diff --git a/src/ _generated_/rest/api/resources/audit/index.ts b/flowvault/_generated_/rest/src/api/resources/records/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/index.ts rename to flowvault/_generated_/rest/src/api/resources/records/index.ts diff --git a/flowvault/_generated_/rest/src/api/resources/records/types/UpdateRequestUpdateType.ts b/flowvault/_generated_/rest/src/api/resources/records/types/UpdateRequestUpdateType.ts new file mode 100644 index 00000000..626c5af8 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/types/UpdateRequestUpdateType.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of update operation to perform. + */ +export type UpdateRequestUpdateType = "UPDATE" | "REPLACE"; +export const UpdateRequestUpdateType = { + Update: "UPDATE", + Replace: "REPLACE", +} as const; diff --git a/flowvault/_generated_/rest/src/api/resources/records/types/index.ts b/flowvault/_generated_/rest/src/api/resources/records/types/index.ts new file mode 100644 index 00000000..8b874903 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/records/types/index.ts @@ -0,0 +1 @@ +export * from "./UpdateRequestUpdateType"; diff --git a/flowvault/_generated_/rest/src/api/resources/tokens/client/Client.ts b/flowvault/_generated_/rest/src/api/resources/tokens/client/Client.ts new file mode 100644 index 00000000..a1325a11 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/tokens/client/Client.ts @@ -0,0 +1,265 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace Tokens { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class Tokens { + constructor(protected readonly _options: Tokens.Options = {}) {} + + /** + * Returns the deterministic token previously issued for each supplied plaintext value within the specified token group. Only applicable to deterministic tokengroups. + * + * @param {Skyflow.GetTokensFromValuesRequest} request + * @param {Tokens.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.ForbiddenError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.TooManyRequestsError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.tokens.getTokens({ + * vaultID: "d408485953784308a000f8dcf81901ef", + * records: [] + * }) + */ + public getTokens( + request: Skyflow.GetTokensFromValuesRequest, + requestOptions?: Tokens.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__getTokens(request, requestOptions)); + } + + private async __getTokens( + request: Skyflow.GetTokensFromValuesRequest, + requestOptions?: Tokens.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/records/getTokens", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.GetTokensFromValuesResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 403: + throw new Skyflow.ForbiddenError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 429: + throw new Skyflow.TooManyRequestsError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/records/getTokens."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Returns values that correspond to the specified tokens. + * + * @param {Skyflow.DetokenizeRequest} request + * @param {Tokens.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.tokens.detokenize({ + * vaultID: "d408485953784308a000f8dcf81901ef", + * tokens: ["RYtQoeJdSQ", "8bd036b6-8fe0-4176-945b-3a5a63e8fd18"] + * }) + */ + public detokenize( + request: Skyflow.DetokenizeRequest, + requestOptions?: Tokens.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__detokenize(request, requestOptions)); + } + + private async __detokenize( + request: Skyflow.DetokenizeRequest, + requestOptions?: Tokens.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/tokens/detokenize", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.DetokenizeResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/tokens/detokenize."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/src/ _generated_/rest/api/resources/records/client/index.ts b/flowvault/_generated_/rest/src/api/resources/tokens/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/index.ts rename to flowvault/_generated_/rest/src/api/resources/tokens/client/index.ts diff --git a/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/DetokenizeRequest.ts b/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/DetokenizeRequest.ts new file mode 100644 index 00000000..0e0dacec --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/DetokenizeRequest.ts @@ -0,0 +1,21 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef", + * tokens: ["RYtQoeJdSQ", "8bd036b6-8fe0-4176-945b-3a5a63e8fd18"] + * } + */ +export interface DetokenizeRequest { + /** ID of the vault. */ + vaultID: string; + /** Token to be detokenized */ + tokens: string[]; + /** List of token groups to redact. */ + tokenGroupRedactions?: Skyflow.TokenGroupRedactions[]; +} diff --git a/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/GetTokensFromValuesRequest.ts b/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/GetTokensFromValuesRequest.ts new file mode 100644 index 00000000..8b9a0d52 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/GetTokensFromValuesRequest.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultID: "d408485953784308a000f8dcf81901ef", + * records: [] + * } + */ +export interface GetTokensFromValuesRequest { + /** ID of the vault. */ + vaultID: string; + /** Array of value/token-group pairs to look up. */ + records: Skyflow.GetTokensFromValuesRequestObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/index.ts b/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/index.ts new file mode 100644 index 00000000..82a675c6 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/resources/tokens/client/requests/index.ts @@ -0,0 +1,2 @@ +export { type GetTokensFromValuesRequest } from "./GetTokensFromValuesRequest"; +export { type DetokenizeRequest } from "./DetokenizeRequest"; diff --git a/src/ _generated_/rest/api/resources/query/index.ts b/flowvault/_generated_/rest/src/api/resources/tokens/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/query/index.ts rename to flowvault/_generated_/rest/src/api/resources/tokens/index.ts diff --git a/flowvault/_generated_/rest/src/api/types/ColumnRedactions.ts b/flowvault/_generated_/rest/src/api/types/ColumnRedactions.ts new file mode 100644 index 00000000..7ab8d8f7 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/ColumnRedactions.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface ColumnRedactions { + /** Name of the column to redact. */ + columnName: string; + /** Name of the redaction type. */ + redaction: string; +} diff --git a/flowvault/_generated_/rest/src/api/types/DeleteResponse.ts b/flowvault/_generated_/rest/src/api/types/DeleteResponse.ts new file mode 100644 index 00000000..947b5018 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/DeleteResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface DeleteResponse { + /** List of deleted records with skyflow ID and any partial errors. */ + records: Skyflow.DeleteResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/DeleteResponseObject.ts b/flowvault/_generated_/rest/src/api/types/DeleteResponseObject.ts new file mode 100644 index 00000000..56ad6d73 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/DeleteResponseObject.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface DeleteResponseObject { + /** Skyflow ID of the deleted record. */ + skyflowID: string; + /** Error message, if any. */ + error?: string; + /** HTTP status code of the response. */ + httpCode: number; +} diff --git a/flowvault/_generated_/rest/src/api/types/DetokenizeResponse.ts b/flowvault/_generated_/rest/src/api/types/DetokenizeResponse.ts new file mode 100644 index 00000000..4fdde282 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/DetokenizeResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface DetokenizeResponse { + /** Detokenized data */ + response: Skyflow.DetokenizeResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/DetokenizeResponseObject.ts b/flowvault/_generated_/rest/src/api/types/DetokenizeResponseObject.ts new file mode 100644 index 00000000..385efc65 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/DetokenizeResponseObject.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface DetokenizeResponseObject { + /** Token that was detokenized. */ + token: string; + value?: Skyflow.GoogleProtobufValue | undefined; + /** Name of the token group. */ + tokenGroupName?: string; + /** Error message, if any. */ + error?: string; + /** HTTP status code of the response. */ + httpCode?: number; + /** Additional metadata associated with the token, such as tableName or skyflowID. */ + metadata?: Record; +} diff --git a/src/ _generated_/rest/api/types/ErrorResponse.ts b/flowvault/_generated_/rest/src/api/types/ErrorResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/ErrorResponse.ts rename to flowvault/_generated_/rest/src/api/types/ErrorResponse.ts diff --git a/flowvault/_generated_/rest/src/api/types/ErrorResponseError.ts b/flowvault/_generated_/rest/src/api/types/ErrorResponseError.ts new file mode 100644 index 00000000..68e842c9 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/ErrorResponseError.ts @@ -0,0 +1,14 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface ErrorResponseError { + /** gRPC status codes. See https://grpc.io/docs/guides/status-codes. */ + grpc_code: number; + http_code: Skyflow.HttpCode; + http_status: string; + message: string; + details?: Record[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/ExecuteQueryRecordResponse.ts b/flowvault/_generated_/rest/src/api/types/ExecuteQueryRecordResponse.ts new file mode 100644 index 00000000..09cf4037 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/ExecuteQueryRecordResponse.ts @@ -0,0 +1,8 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface ExecuteQueryRecordResponse { + /** Fields and values for the record. */ + data?: Record; +} diff --git a/flowvault/_generated_/rest/src/api/types/ExecuteQueryResponse.ts b/flowvault/_generated_/rest/src/api/types/ExecuteQueryResponse.ts new file mode 100644 index 00000000..71321863 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/ExecuteQueryResponse.ts @@ -0,0 +1,11 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface ExecuteQueryResponse { + /** Records corresponding to the specified query. */ + records?: Skyflow.ExecuteQueryRecordResponse[]; + metadata?: Skyflow.ExecuteQueryResponseMetadata; +} diff --git a/flowvault/_generated_/rest/src/api/types/ExecuteQueryResponseMetadata.ts b/flowvault/_generated_/rest/src/api/types/ExecuteQueryResponseMetadata.ts new file mode 100644 index 00000000..1237e10b --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/ExecuteQueryResponseMetadata.ts @@ -0,0 +1,11 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Metadata for the query. + */ +export interface ExecuteQueryResponseMetadata { + /** Columns returned for the query. */ + columns?: string[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/FileDeleteRecord.ts b/flowvault/_generated_/rest/src/api/types/FileDeleteRecord.ts new file mode 100644 index 00000000..52b27fa3 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/FileDeleteRecord.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface FileDeleteRecord { + /** Name of the table. */ + tableName: string; + /** Skyflow ID of the record containing the file column(s) to delete. Either skyflowID or uniqueValues must be specified, not both. */ + skyflowID?: string; + /** Names of the file columns to delete on this record. If any listed column has no file present, the request fails for this record. */ + columns: string[]; + /** List of unique constraint values used to look up the record(s) to delete, in place of skyflowID. If more than one object is provided, each is resolved independently and the specified columns are deleted from every matching record. */ + uniqueValues?: Skyflow.UniqueValue[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/FileDeleteResponse.ts b/flowvault/_generated_/rest/src/api/types/FileDeleteResponse.ts new file mode 100644 index 00000000..65be3f92 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/FileDeleteResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface FileDeleteResponse { + /** List of records (max 100) with the result of the delete operation. */ + records: Skyflow.FileDeleteResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/FileDeleteResponseObject.ts b/flowvault/_generated_/rest/src/api/types/FileDeleteResponseObject.ts new file mode 100644 index 00000000..8dfbd5cc --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/FileDeleteResponseObject.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface FileDeleteResponseObject { + /** Skyflow ID of the record that was targeted. When the request specifies uniqueValues, this is the skyflowID resolved from the matching record. Null if the record failed validation before a skyflowID could be resolved. */ + skyflowID?: string; + /** Name of the table. */ + tableName: string; + /** Map of column name to deletion result. Populated only when the record's delete succeeds, with one entry per requested column with status DELETED. Null if any requested column has no file present, in which case error is populated instead. */ + data?: Record; + /** Error message describing any issue with the specific record. Null if no error occurred. */ + error?: string; + /** HTTP status code representing the result of the API request for this record. */ + httpCode: number; +} diff --git a/flowvault/_generated_/rest/src/api/types/FileUploadColumn.ts b/flowvault/_generated_/rest/src/api/types/FileUploadColumn.ts new file mode 100644 index 00000000..5fe790de --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/FileUploadColumn.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface FileUploadColumn { + /** Name of the column where the file is being uploaded. */ + column: string; + /** Name of the file being uploaded to the file column. If not provided, a random 16-byte UUID name is generated. */ + fileName?: string; +} diff --git a/flowvault/_generated_/rest/src/api/types/FileUploadRecord.ts b/flowvault/_generated_/rest/src/api/types/FileUploadRecord.ts new file mode 100644 index 00000000..7283fafc --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/FileUploadRecord.ts @@ -0,0 +1,14 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface FileUploadRecord { + /** Name of the table. */ + tableName: string; + /** Skyflow ID of the record. If not provided, a new record is created and its skyflowID is used for the file upload. */ + skyflowID?: string; + /** List of file columns to generate signed upload URLs for. */ + columns: Skyflow.FileUploadColumn[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/FileUploadResponse.ts b/flowvault/_generated_/rest/src/api/types/FileUploadResponse.ts new file mode 100644 index 00000000..92aafd48 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/FileUploadResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface FileUploadResponse { + /** List of records with signed upload URLs. */ + records: Skyflow.FileUploadResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/FileUploadResponseObject.ts b/flowvault/_generated_/rest/src/api/types/FileUploadResponseObject.ts new file mode 100644 index 00000000..413e0e67 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/FileUploadResponseObject.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface FileUploadResponseObject { + /** Skyflow ID of the record. A new skyflowID is returned if one was not provided in the request. */ + skyflowID: string; + /** Name of the table. */ + tableName: string; + /** Map of column names to signed upload URLs. Each signed URL expires after 15 minutes by default. */ + data?: Record; + /** Error message describing any issue with the specific record. Null if no error occurred. */ + error?: string; + /** HTTP status code representing the result of the API request for this record. */ + httpCode: number; +} diff --git a/flowvault/_generated_/rest/src/api/types/GetRequestData.ts b/flowvault/_generated_/rest/src/api/types/GetRequestData.ts new file mode 100644 index 00000000..f1fa4f02 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/GetRequestData.ts @@ -0,0 +1,18 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface GetRequestData { + /** Name of the table. */ + tableName: string; + /** Skyflow IDs of the records to return. */ + skyflowIDs: string[]; + /** List of columns to redact. */ + columnRedactions?: Skyflow.ColumnRedactions[]; + /** List of columns to return. */ + columns?: string[]; + /** List of unique constraint values to query records by data. */ + uniqueValues?: Skyflow.UniqueValue[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/GetResponse.ts b/flowvault/_generated_/rest/src/api/types/GetResponse.ts new file mode 100644 index 00000000..51cb78b7 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/GetResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface GetResponse { + /** List of fetched records. For file columns, the value in the data map is an object containing: fileName, mimeType, sizeBytes, fileStatus (PENDING | READY | FAILED | SCAN_ERROR), fileFailureReason (non-null only when fileStatus is FAILED), url (pre-signed download URL, non-null only when fileStatus is READY), and urlExpiresAt (UTC expiry of the URL). */ + records: Skyflow.RecordResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/GetTokensFromValuesRequestObject.ts b/flowvault/_generated_/rest/src/api/types/GetTokensFromValuesRequestObject.ts new file mode 100644 index 00000000..dfa91a91 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/GetTokensFromValuesRequestObject.ts @@ -0,0 +1,11 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface GetTokensFromValuesRequestObject { + value?: Skyflow.GoogleProtobufValue; + /** Name of the deterministic token group. */ + tokenGroupName: string; +} diff --git a/flowvault/_generated_/rest/src/api/types/GetTokensFromValuesResponse.ts b/flowvault/_generated_/rest/src/api/types/GetTokensFromValuesResponse.ts new file mode 100644 index 00000000..3a11cf9d --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/GetTokensFromValuesResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface GetTokensFromValuesResponse { + /** Array of token result objects, one per input entry, in the same order as the request. */ + records: Skyflow.TokenizeResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/GoogleProtobufValue.ts b/flowvault/_generated_/rest/src/api/types/GoogleProtobufValue.ts new file mode 100644 index 00000000..d3d90c4c --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/GoogleProtobufValue.ts @@ -0,0 +1,8 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Represents a dynamically typed value which can be either null, a number, a string, a boolean, a recursive struct value, or a list of values. + */ +export type GoogleProtobufValue = unknown; diff --git a/flowvault/_generated_/rest/src/api/types/HttpCode.ts b/flowvault/_generated_/rest/src/api/types/HttpCode.ts new file mode 100644 index 00000000..e85dbda8 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/HttpCode.ts @@ -0,0 +1,8 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * HTTP status codes. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Status. + */ +export type HttpCode = number; diff --git a/flowvault/_generated_/rest/src/api/types/InsertRecordData.ts b/flowvault/_generated_/rest/src/api/types/InsertRecordData.ts new file mode 100644 index 00000000..dc23f38c --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/InsertRecordData.ts @@ -0,0 +1,15 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface InsertRecordData { + /** Columns and values for the record. */ + data: Record; + /** Columns and tokens for the record. */ + tokens?: Record; + /** Name of the table to insert data into. */ + tableName?: string; + upsert?: Skyflow.Upsert; +} diff --git a/flowvault/_generated_/rest/src/api/types/InsertResponse.ts b/flowvault/_generated_/rest/src/api/types/InsertResponse.ts new file mode 100644 index 00000000..02fec926 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/InsertResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface InsertResponse { + /** List of inserted records. */ + records: Skyflow.RecordResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/RecordResponseObject.ts b/flowvault/_generated_/rest/src/api/types/RecordResponseObject.ts new file mode 100644 index 00000000..376bd587 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/RecordResponseObject.ts @@ -0,0 +1,20 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface RecordResponseObject { + /** Skyflow ID for the inserted record */ + skyflowID?: string | null; + /** Columns and tokens for the record. */ + tokens?: Record; + /** Columns and values for the record. For file columns, the value is an object containing file metadata: fileName, mimeType, sizeBytes, fileStatus (PENDING | READY | FAILED | SCAN_ERROR), fileFailureReason (populated only when fileStatus is FAILED), url (pre-signed download URL, populated only when fileStatus is READY), and urlExpiresAt (UTC expiry of the URL). */ + data?: Record; + /** Columns and hashed values for the record. */ + hashedData?: Record; + /** Error message, if any. */ + error?: string | null; + /** HTTP status code of the response. */ + httpCode: number; + /** Name of the table that the record belongs to. */ + tableName?: string; +} diff --git a/flowvault/_generated_/rest/src/api/types/TokenGroupRedactions.ts b/flowvault/_generated_/rest/src/api/types/TokenGroupRedactions.ts new file mode 100644 index 00000000..a1cb1236 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/TokenGroupRedactions.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface TokenGroupRedactions { + /** Name of the token group to redact. */ + tokenGroupName?: string; + /** Name of the redaction to perform. */ + redaction?: string; +} diff --git a/flowvault/_generated_/rest/src/api/types/TokenizeResponseObject.ts b/flowvault/_generated_/rest/src/api/types/TokenizeResponseObject.ts new file mode 100644 index 00000000..74c20845 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/TokenizeResponseObject.ts @@ -0,0 +1,17 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface TokenizeResponseObject { + /** Token that was generated. */ + token: string; + value?: Skyflow.GoogleProtobufValue | undefined; + /** Name of the token group. */ + tokenGroupName: string | null; + /** Error message, if any. */ + error?: string; + /** HTTP status code of the response. */ + httpCode?: number; +} diff --git a/flowvault/_generated_/rest/src/api/types/UniqueValue.ts b/flowvault/_generated_/rest/src/api/types/UniqueValue.ts new file mode 100644 index 00000000..2a0e2356 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/UniqueValue.ts @@ -0,0 +1,8 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface UniqueValue { + /** Columns names and values for the unique value entry. */ + data: Record; +} diff --git a/flowvault/_generated_/rest/src/api/types/UpdateRecordData.ts b/flowvault/_generated_/rest/src/api/types/UpdateRecordData.ts new file mode 100644 index 00000000..92e97383 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/UpdateRecordData.ts @@ -0,0 +1,18 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface UpdateRecordData { + /** Skyflow ID of the record to update. */ + skyflowID: string; + /** Columns and values for the record. */ + data: Record; + /** Columns and tokens for the record. */ + tokens?: Record; + /** Name of the table to update data in. */ + tableName?: string; + /** Type of update operation to perform. */ + updateType?: Skyflow.UpdateRecordDataUpdateType; +} diff --git a/flowvault/_generated_/rest/src/api/types/UpdateRecordDataUpdateType.ts b/flowvault/_generated_/rest/src/api/types/UpdateRecordDataUpdateType.ts new file mode 100644 index 00000000..dc4b89a4 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/UpdateRecordDataUpdateType.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of update operation to perform. + */ +export type UpdateRecordDataUpdateType = "UPDATE" | "REPLACE"; +export const UpdateRecordDataUpdateType = { + Update: "UPDATE", + Replace: "REPLACE", +} as const; diff --git a/flowvault/_generated_/rest/src/api/types/UpdateResponse.ts b/flowvault/_generated_/rest/src/api/types/UpdateResponse.ts new file mode 100644 index 00000000..ee933ba2 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/UpdateResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface UpdateResponse { + /** List of updated records. */ + records: Skyflow.RecordResponseObject[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/Upsert.ts b/flowvault/_generated_/rest/src/api/types/Upsert.ts new file mode 100644 index 00000000..4af2b7af --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/Upsert.ts @@ -0,0 +1,15 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Upsert details. + */ +export interface Upsert { + /** Type of update operation to perform. */ + updateType?: Skyflow.UpsertUpdateType; + /** List of unique columns in the table that upsert operations use to identify if a record with matching values exists. If a matching record exists, the record updates with the specified values. If a matching record doesn't exist, the upsert operation inserts a new record. */ + uniqueColumns: string[]; +} diff --git a/flowvault/_generated_/rest/src/api/types/UpsertUpdateType.ts b/flowvault/_generated_/rest/src/api/types/UpsertUpdateType.ts new file mode 100644 index 00000000..07929246 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/UpsertUpdateType.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of update operation to perform. + */ +export type UpsertUpdateType = "UPDATE" | "REPLACE"; +export const UpsertUpdateType = { + Update: "UPDATE", + Replace: "REPLACE", +} as const; diff --git a/flowvault/_generated_/rest/src/api/types/index.ts b/flowvault/_generated_/rest/src/api/types/index.ts new file mode 100644 index 00000000..0ca892a1 --- /dev/null +++ b/flowvault/_generated_/rest/src/api/types/index.ts @@ -0,0 +1,34 @@ +export * from "./ColumnRedactions"; +export * from "./DeleteResponse"; +export * from "./DeleteResponseObject"; +export * from "./ExecuteQueryRecordResponse"; +export * from "./ExecuteQueryResponse"; +export * from "./ExecuteQueryResponseMetadata"; +export * from "./FileDeleteRecord"; +export * from "./FileDeleteResponse"; +export * from "./FileDeleteResponseObject"; +export * from "./FileUploadColumn"; +export * from "./FileUploadRecord"; +export * from "./FileUploadResponse"; +export * from "./FileUploadResponseObject"; +export * from "./DetokenizeResponse"; +export * from "./DetokenizeResponseObject"; +export * from "./GetTokensFromValuesRequestObject"; +export * from "./GetTokensFromValuesResponse"; +export * from "./TokenizeResponseObject"; +export * from "./GetRequestData"; +export * from "./GetResponse"; +export * from "./GoogleProtobufValue"; +export * from "./InsertRecordData"; +export * from "./InsertResponse"; +export * from "./RecordResponseObject"; +export * from "./TokenGroupRedactions"; +export * from "./UniqueValue"; +export * from "./UpdateRecordDataUpdateType"; +export * from "./UpdateRecordData"; +export * from "./UpdateResponse"; +export * from "./UpsertUpdateType"; +export * from "./Upsert"; +export * from "./HttpCode"; +export * from "./ErrorResponseError"; +export * from "./ErrorResponse"; diff --git a/flowvault/_generated_/rest/src/core/auth/BasicAuth.ts b/flowvault/_generated_/rest/src/core/auth/BasicAuth.ts new file mode 100644 index 00000000..146df215 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/auth/BasicAuth.ts @@ -0,0 +1,31 @@ +import { Base64 } from "js-base64"; + +export interface BasicAuth { + username: string; + password: string; +} + +const BASIC_AUTH_HEADER_PREFIX = /^Basic /i; + +export const BasicAuth = { + toAuthorizationHeader: (basicAuth: BasicAuth | undefined): string | undefined => { + if (basicAuth == null) { + return undefined; + } + const token = Base64.encode(`${basicAuth.username}:${basicAuth.password}`); + return `Basic ${token}`; + }, + fromAuthorizationHeader: (header: string): BasicAuth => { + const credentials = header.replace(BASIC_AUTH_HEADER_PREFIX, ""); + const decoded = Base64.decode(credentials); + const [username, password] = decoded.split(":", 2); + + if (username == null || password == null) { + throw new Error("Invalid basic auth"); + } + return { + username, + password, + }; + }, +}; diff --git a/flowvault/_generated_/rest/src/core/auth/BearerToken.ts b/flowvault/_generated_/rest/src/core/auth/BearerToken.ts new file mode 100644 index 00000000..fe987fc9 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/auth/BearerToken.ts @@ -0,0 +1,15 @@ +export type BearerToken = string; + +const BEARER_AUTH_HEADER_PREFIX = /^Bearer /i; + +export const BearerToken = { + toAuthorizationHeader: (token: BearerToken | undefined): string | undefined => { + if (token == null) { + return undefined; + } + return `Bearer ${token}`; + }, + fromAuthorizationHeader: (header: string): BearerToken => { + return header.replace(BEARER_AUTH_HEADER_PREFIX, "").trim() as BearerToken; + }, +}; diff --git a/flowvault/_generated_/rest/src/core/auth/index.ts b/flowvault/_generated_/rest/src/core/auth/index.ts new file mode 100644 index 00000000..ee293b34 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/auth/index.ts @@ -0,0 +1,2 @@ +export { BasicAuth } from "./BasicAuth"; +export { BearerToken } from "./BearerToken"; diff --git a/flowvault/_generated_/rest/src/core/fetcher/APIResponse.ts b/flowvault/_generated_/rest/src/core/fetcher/APIResponse.ts new file mode 100644 index 00000000..6335291b --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/APIResponse.ts @@ -0,0 +1,23 @@ +import { RawResponse } from "./RawResponse"; + +/** + * The response of an API call. + * It is a successful response or a failed response. + */ +export type APIResponse = SuccessfulResponse | FailedResponse; + +export interface SuccessfulResponse { + ok: true; + body: T; + /** + * @deprecated Use `rawResponse` instead + */ + headers?: Record; + rawResponse: RawResponse; +} + +export interface FailedResponse { + ok: false; + error: T; + rawResponse: RawResponse; +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/Fetcher.ts b/flowvault/_generated_/rest/src/core/fetcher/Fetcher.ts new file mode 100644 index 00000000..336ee10b --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/Fetcher.ts @@ -0,0 +1,151 @@ +import { toJson } from "../json"; +import { APIResponse } from "./APIResponse"; +import { abortRawResponse, toRawResponse, unknownRawResponse } from "./RawResponse"; +import { createRequestUrl } from "./createRequestUrl"; +import { getFetchFn } from "./getFetchFn"; +import { getRequestBody } from "./getRequestBody"; +import { getResponseBody } from "./getResponseBody"; +import { makeRequest } from "./makeRequest"; +import { requestWithRetries } from "./requestWithRetries"; + +export type FetchFunction = (args: Fetcher.Args) => Promise>; + +export declare namespace Fetcher { + export interface Args { + url: string; + method: string; + contentType?: string; + headers?: Record; + queryParameters?: Record; + body?: unknown; + timeoutMs?: number; + maxRetries?: number; + withCredentials?: boolean; + abortSignal?: AbortSignal; + requestType?: "json" | "file" | "bytes"; + responseType?: "json" | "blob" | "sse" | "streaming" | "text" | "arrayBuffer"; + duplex?: "half"; + } + + export type Error = FailedStatusCodeError | NonJsonError | TimeoutError | UnknownError; + + export interface FailedStatusCodeError { + reason: "status-code"; + statusCode: number; + body: unknown; + } + + export interface NonJsonError { + reason: "non-json"; + statusCode: number; + rawBody: string; + } + + export interface TimeoutError { + reason: "timeout"; + } + + export interface UnknownError { + reason: "unknown"; + errorMessage: string; + } +} + +export async function fetcherImpl(args: Fetcher.Args): Promise> { + const headers: Record = {}; + if (args.body !== undefined && args.contentType != null) { + headers["Content-Type"] = args.contentType; + } + + if (args.headers != null) { + for (const [key, value] of Object.entries(args.headers)) { + if (value != null) { + headers[key] = value; + } + } + } + + const url = createRequestUrl(args.url, args.queryParameters); + const requestBody: BodyInit | undefined = await getRequestBody({ + body: args.body, + type: args.requestType === "json" ? "json" : "other", + }); + const fetchFn = await getFetchFn(); + + try { + const response = await requestWithRetries( + async () => + makeRequest( + fetchFn, + url, + args.method, + headers, + requestBody, + args.timeoutMs, + args.abortSignal, + args.withCredentials, + args.duplex, + ), + args.maxRetries, + ); + const responseBody = await getResponseBody(response, args.responseType); + + if (response.status >= 200 && response.status < 400) { + return { + ok: true, + body: responseBody as R, + headers: response.headers, + rawResponse: toRawResponse(response), + }; + } else { + return { + ok: false, + error: { + reason: "status-code", + statusCode: response.status, + body: responseBody, + }, + rawResponse: toRawResponse(response), + }; + } + } catch (error) { + if (args.abortSignal != null && args.abortSignal.aborted) { + return { + ok: false, + error: { + reason: "unknown", + errorMessage: "The user aborted a request", + }, + rawResponse: abortRawResponse, + }; + } else if (error instanceof Error && error.name === "AbortError") { + return { + ok: false, + error: { + reason: "timeout", + }, + rawResponse: abortRawResponse, + }; + } else if (error instanceof Error) { + return { + ok: false, + error: { + reason: "unknown", + errorMessage: error.message, + }, + rawResponse: unknownRawResponse, + }; + } + + return { + ok: false, + error: { + reason: "unknown", + errorMessage: toJson(error), + }, + rawResponse: unknownRawResponse, + }; + } +} + +export const fetcher: FetchFunction = fetcherImpl; diff --git a/flowvault/_generated_/rest/src/core/fetcher/HttpResponsePromise.ts b/flowvault/_generated_/rest/src/core/fetcher/HttpResponsePromise.ts new file mode 100644 index 00000000..6b0bc4fc --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/HttpResponsePromise.ts @@ -0,0 +1,116 @@ +import { WithRawResponse } from "./RawResponse"; + +/** + * A promise that returns the parsed response and lets you retrieve the raw response too. + */ +export class HttpResponsePromise extends Promise { + private innerPromise: Promise>; + private unwrappedPromise: Promise | undefined; + + private constructor(promise: Promise>) { + // Initialize with a no-op to avoid premature parsing + super((resolve) => { + resolve(undefined as unknown as T); + }); + this.innerPromise = promise; + } + + /** + * Creates an `HttpResponsePromise` from a function that returns a promise. + * + * @param fn - A function that returns a promise resolving to a `WithRawResponse` object. + * @param args - Arguments to pass to the function. + * @returns An `HttpResponsePromise` instance. + */ + public static fromFunction Promise>, T>( + fn: F, + ...args: Parameters + ): HttpResponsePromise { + return new HttpResponsePromise(fn(...args)); + } + + /** + * Creates a function that returns an `HttpResponsePromise` from a function that returns a promise. + * + * @param fn - A function that returns a promise resolving to a `WithRawResponse` object. + * @returns A function that returns an `HttpResponsePromise` instance. + */ + public static interceptFunction< + F extends (...args: never[]) => Promise>, + T = Awaited>["data"], + >(fn: F): (...args: Parameters) => HttpResponsePromise { + return (...args: Parameters): HttpResponsePromise => { + return HttpResponsePromise.fromPromise(fn(...args)); + }; + } + + /** + * Creates an `HttpResponsePromise` from an existing promise. + * + * @param promise - A promise resolving to a `WithRawResponse` object. + * @returns An `HttpResponsePromise` instance. + */ + public static fromPromise(promise: Promise>): HttpResponsePromise { + return new HttpResponsePromise(promise); + } + + /** + * Creates an `HttpResponsePromise` from an executor function. + * + * @param executor - A function that takes resolve and reject callbacks to create a promise. + * @returns An `HttpResponsePromise` instance. + */ + public static fromExecutor( + executor: (resolve: (value: WithRawResponse) => void, reject: (reason?: unknown) => void) => void, + ): HttpResponsePromise { + const promise = new Promise>(executor); + return new HttpResponsePromise(promise); + } + + /** + * Creates an `HttpResponsePromise` from a resolved result. + * + * @param result - A `WithRawResponse` object to resolve immediately. + * @returns An `HttpResponsePromise` instance. + */ + public static fromResult(result: WithRawResponse): HttpResponsePromise { + const promise = Promise.resolve(result); + return new HttpResponsePromise(promise); + } + + private unwrap(): Promise { + if (!this.unwrappedPromise) { + this.unwrappedPromise = this.innerPromise.then(({ data }) => data); + } + return this.unwrappedPromise; + } + + /** @inheritdoc */ + public override then( + onfulfilled?: ((value: T) => TResult1 | PromiseLike) | null, + onrejected?: ((reason: unknown) => TResult2 | PromiseLike) | null, + ): Promise { + return this.unwrap().then(onfulfilled, onrejected); + } + + /** @inheritdoc */ + public override catch( + onrejected?: ((reason: unknown) => TResult | PromiseLike) | null, + ): Promise { + return this.unwrap().catch(onrejected); + } + + /** @inheritdoc */ + public override finally(onfinally?: (() => void) | null): Promise { + return this.unwrap().finally(onfinally); + } + + /** + * Retrieves the data and raw response. + * + * @returns A promise resolving to a `WithRawResponse` object. + */ + public async withRawResponse(): Promise> { + return await this.innerPromise; + } +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/RawResponse.ts b/flowvault/_generated_/rest/src/core/fetcher/RawResponse.ts new file mode 100644 index 00000000..c975f23c --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/RawResponse.ts @@ -0,0 +1,59 @@ +/** + * The raw response from the fetch call excluding the body. + */ +export type RawResponse = Omit< + { + [K in keyof Response as Response[K] extends Function ? never : K]: Response[K]; // strips out functions + }, + "ok" | "body" | "bodyUsed" +>; // strips out body and bodyUsed + +/** + * A raw response indicating that the request was aborted. + */ +export const abortRawResponse: RawResponse = { + headers: new Headers(), + redirected: false, + status: 499, + statusText: "Client Closed Request", + type: "error", + url: "", +} as const; + +/** + * A raw response indicating an unknown error. + */ +export const unknownRawResponse: RawResponse = { + headers: new Headers(), + redirected: false, + status: 0, + statusText: "Unknown Error", + type: "error", + url: "", +} as const; + +/** + * Converts a `RawResponse` object into a `RawResponse` by extracting its properties, + * excluding the `body` and `bodyUsed` fields. + * + * @param response - The `RawResponse` object to convert. + * @returns A `RawResponse` object containing the extracted properties of the input response. + */ +export function toRawResponse(response: Response): RawResponse { + return { + headers: response.headers, + redirected: response.redirected, + status: response.status, + statusText: response.statusText, + type: response.type, + url: response.url, + }; +} + +/** + * Creates a `RawResponse` from a standard `Response` object. + */ +export interface WithRawResponse { + readonly data: T; + readonly rawResponse: RawResponse; +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/Supplier.ts b/flowvault/_generated_/rest/src/core/fetcher/Supplier.ts new file mode 100644 index 00000000..867c931c --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/Supplier.ts @@ -0,0 +1,11 @@ +export type Supplier = T | Promise | (() => T | Promise); + +export const Supplier = { + get: async (supplier: Supplier): Promise => { + if (typeof supplier === "function") { + return (supplier as () => T)(); + } else { + return supplier; + } + }, +}; diff --git a/flowvault/_generated_/rest/src/core/fetcher/createRequestUrl.ts b/flowvault/_generated_/rest/src/core/fetcher/createRequestUrl.ts new file mode 100644 index 00000000..f1157ce7 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/createRequestUrl.ts @@ -0,0 +1,10 @@ +import qs from "qs"; + +export function createRequestUrl( + baseUrl: string, + queryParameters?: Record, +): string { + return Object.keys(queryParameters ?? {}).length > 0 + ? `${baseUrl}?${qs.stringify(queryParameters, { arrayFormat: "repeat" })}` + : baseUrl; +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/getFetchFn.ts b/flowvault/_generated_/rest/src/core/fetcher/getFetchFn.ts new file mode 100644 index 00000000..9fd9bfc4 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/getFetchFn.ts @@ -0,0 +1,25 @@ +import { RUNTIME } from "../runtime"; + +/** + * Returns a fetch function based on the runtime + */ +export async function getFetchFn(): Promise { + // In Node.js 18+ environments, use native fetch + if (RUNTIME.type === "node" && RUNTIME.parsedVersion != null && RUNTIME.parsedVersion >= 18) { + return fetch; + } + + // In Node.js 18 or lower environments, the SDK always uses`node-fetch`. + if (RUNTIME.type === "node") { + return (await import("node-fetch")).default as any; + } + + // Otherwise the SDK uses global fetch if available, + // and falls back to node-fetch. + if (typeof fetch == "function") { + return fetch; + } + + // Defaults to node `node-fetch` if global fetch isn't available + return (await import("node-fetch")).default as any; +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/getHeader.ts b/flowvault/_generated_/rest/src/core/fetcher/getHeader.ts new file mode 100644 index 00000000..50f922b0 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/getHeader.ts @@ -0,0 +1,8 @@ +export function getHeader(headers: Record, header: string): string | undefined { + for (const [headerKey, headerValue] of Object.entries(headers)) { + if (headerKey.toLowerCase() === header.toLowerCase()) { + return headerValue; + } + } + return undefined; +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/getRequestBody.ts b/flowvault/_generated_/rest/src/core/fetcher/getRequestBody.ts new file mode 100644 index 00000000..fce5589c --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/getRequestBody.ts @@ -0,0 +1,16 @@ +import { toJson } from "../json"; + +export declare namespace GetRequestBody { + interface Args { + body: unknown; + type: "json" | "file" | "bytes" | "other"; + } +} + +export async function getRequestBody({ body, type }: GetRequestBody.Args): Promise { + if (type.includes("json")) { + return toJson(body); + } else { + return body as BodyInit; + } +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/getResponseBody.ts b/flowvault/_generated_/rest/src/core/fetcher/getResponseBody.ts new file mode 100644 index 00000000..d046e6ea --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/getResponseBody.ts @@ -0,0 +1,34 @@ +import { chooseStreamWrapper } from "./stream-wrappers/chooseStreamWrapper"; + +export async function getResponseBody(response: Response, responseType?: string): Promise { + if (response.body != null && responseType === "blob") { + return await response.blob(); + } else if (response.body != null && responseType === "arrayBuffer") { + return await response.arrayBuffer(); + } else if (response.body != null && responseType === "sse") { + return response.body; + } else if (response.body != null && responseType === "streaming") { + return chooseStreamWrapper(response.body); + } else if (response.body != null && responseType === "text") { + return await response.text(); + } else { + const text = await response.text(); + if (text.length > 0) { + try { + let responseBody = JSON.parse(text); + return responseBody; + } catch (err) { + return { + ok: false, + error: { + reason: "non-json", + statusCode: response.status, + rawBody: text, + }, + }; + } + } else { + return undefined; + } + } +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/index.ts b/flowvault/_generated_/rest/src/core/fetcher/index.ts new file mode 100644 index 00000000..249f5176 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/index.ts @@ -0,0 +1,8 @@ +export type { APIResponse } from "./APIResponse"; +export { fetcher } from "./Fetcher"; +export type { Fetcher, FetchFunction } from "./Fetcher"; +export { getHeader } from "./getHeader"; +export { Supplier } from "./Supplier"; +export { abortRawResponse, toRawResponse, unknownRawResponse } from "./RawResponse"; +export type { RawResponse, WithRawResponse } from "./RawResponse"; +export { HttpResponsePromise } from "./HttpResponsePromise"; diff --git a/flowvault/_generated_/rest/src/core/fetcher/makeRequest.ts b/flowvault/_generated_/rest/src/core/fetcher/makeRequest.ts new file mode 100644 index 00000000..1af42bb9 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/makeRequest.ts @@ -0,0 +1,44 @@ +import { anySignal, getTimeoutSignal } from "./signals"; + +export const makeRequest = async ( + fetchFn: (url: string, init: RequestInit) => Promise, + url: string, + method: string, + headers: Record, + requestBody: BodyInit | undefined, + timeoutMs?: number, + abortSignal?: AbortSignal, + withCredentials?: boolean, + duplex?: "half", +): Promise => { + const signals: AbortSignal[] = []; + + // Add timeout signal + let timeoutAbortId: NodeJS.Timeout | undefined = undefined; + if (timeoutMs != null) { + const { signal, abortId } = getTimeoutSignal(timeoutMs); + timeoutAbortId = abortId; + signals.push(signal); + } + + // Add arbitrary signal + if (abortSignal != null) { + signals.push(abortSignal); + } + let newSignals = anySignal(signals); + const response = await fetchFn(url, { + method: method, + headers, + body: requestBody, + signal: newSignals, + credentials: withCredentials ? "include" : undefined, + // @ts-ignore + duplex, + }); + + if (timeoutAbortId != null) { + clearTimeout(timeoutAbortId); + } + + return response; +}; diff --git a/flowvault/_generated_/rest/src/core/fetcher/requestWithRetries.ts b/flowvault/_generated_/rest/src/core/fetcher/requestWithRetries.ts new file mode 100644 index 00000000..add3cce0 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/requestWithRetries.ts @@ -0,0 +1,33 @@ +const INITIAL_RETRY_DELAY = 1000; // in milliseconds +const MAX_RETRY_DELAY = 60000; // in milliseconds +const DEFAULT_MAX_RETRIES = 2; +const JITTER_FACTOR = 0.2; // 20% random jitter + +function addJitter(delay: number): number { + // Generate a random value between -JITTER_FACTOR and +JITTER_FACTOR + const jitterMultiplier = 1 + (Math.random() * 2 - 1) * JITTER_FACTOR; + return delay * jitterMultiplier; +} + +export async function requestWithRetries( + requestFn: () => Promise, + maxRetries: number = DEFAULT_MAX_RETRIES, +): Promise { + let response: Response = await requestFn(); + + for (let i = 0; i < maxRetries; ++i) { + if ([408, 429].includes(response.status) || response.status >= 500) { + // Calculate base delay using exponential backoff (in milliseconds) + const baseDelay = Math.min(INITIAL_RETRY_DELAY * Math.pow(2, i), MAX_RETRY_DELAY); + + // Add jitter to the delay + const delayWithJitter = addJitter(baseDelay); + + await new Promise((resolve) => setTimeout(resolve, delayWithJitter)); + response = await requestFn(); + } else { + break; + } + } + return response!; +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/signals.ts b/flowvault/_generated_/rest/src/core/fetcher/signals.ts new file mode 100644 index 00000000..a8d32a2e --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/signals.ts @@ -0,0 +1,38 @@ +const TIMEOUT = "timeout"; + +export function getTimeoutSignal(timeoutMs: number): { signal: AbortSignal; abortId: NodeJS.Timeout } { + const controller = new AbortController(); + const abortId = setTimeout(() => controller.abort(TIMEOUT), timeoutMs); + return { signal: controller.signal, abortId }; +} + +/** + * Returns an abort signal that is getting aborted when + * at least one of the specified abort signals is aborted. + * + * Requires at least node.js 18. + */ +export function anySignal(...args: AbortSignal[] | [AbortSignal[]]): AbortSignal { + // Allowing signals to be passed either as array + // of signals or as multiple arguments. + const signals = (args.length === 1 && Array.isArray(args[0]) ? args[0] : args) as AbortSignal[]; + + const controller = new AbortController(); + + for (const signal of signals) { + if (signal.aborted) { + // Exiting early if one of the signals + // is already aborted. + controller.abort((signal as any)?.reason); + break; + } + + // Listening for signals and removing the listeners + // when at least one symbol is aborted. + signal.addEventListener("abort", () => controller.abort((signal as any)?.reason), { + signal: controller.signal, + }); + } + + return controller.signal; +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts new file mode 100644 index 00000000..7d83aef2 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts @@ -0,0 +1,259 @@ +import type { Writable } from "readable-stream"; + +import { EventCallback, StreamWrapper } from "./chooseStreamWrapper"; + +export class Node18UniversalStreamWrapper< + ReadFormat extends Uint8Array | Uint16Array | Uint32Array, +> implements StreamWrapper< + Node18UniversalStreamWrapper | Writable | WritableStream, + ReadFormat +> { + private readableStream: ReadableStream; + private reader: ReadableStreamDefaultReader; + private events: Record; + private paused: boolean; + private resumeCallback: ((value?: unknown) => void) | null; + private encoding: string | null; + + constructor(readableStream: ReadableStream) { + this.readableStream = readableStream; + this.reader = this.readableStream.getReader(); + this.events = { + data: [], + end: [], + error: [], + readable: [], + close: [], + pause: [], + resume: [], + }; + this.paused = false; + this.resumeCallback = null; + this.encoding = null; + } + + public on(event: string, callback: EventCallback): void { + this.events[event]?.push(callback); + } + + public off(event: string, callback: EventCallback): void { + this.events[event] = this.events[event]?.filter((cb) => cb !== callback); + } + + public pipe( + dest: Node18UniversalStreamWrapper | Writable | WritableStream, + ): Node18UniversalStreamWrapper | Writable | WritableStream { + this.on("data", async (chunk) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._write(chunk); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } else { + dest.write(chunk); + } + }); + + this.on("end", async () => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._end(); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.close(); + } else { + dest.end(); + } + }); + + this.on("error", async (error) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._error(error); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.abort(error); + } else { + dest.destroy(error); + } + }); + + this._startReading(); + + return dest; + } + + public pipeTo( + dest: Node18UniversalStreamWrapper | Writable | WritableStream, + ): Node18UniversalStreamWrapper | Writable | WritableStream { + return this.pipe(dest); + } + + public unpipe(dest: Node18UniversalStreamWrapper | Writable | WritableStream): void { + this.off("data", async (chunk) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._write(chunk); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } else { + dest.write(chunk); + } + }); + + this.off("end", async () => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._end(); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.close(); + } else { + dest.end(); + } + }); + + this.off("error", async (error) => { + if (dest instanceof Node18UniversalStreamWrapper) { + dest._error(error); + } else if (dest instanceof WritableStream) { + const writer = dest.getWriter(); + writer.abort(error); + } else { + dest.destroy(error); + } + }); + } + + public destroy(error?: Error): void { + this.reader + .cancel(error) + .then(() => { + this._emit("close"); + }) + .catch((err) => { + this._emit("error", err); + }); + } + + public pause(): void { + this.paused = true; + this._emit("pause"); + } + + public resume(): void { + if (this.paused) { + this.paused = false; + this._emit("resume"); + if (this.resumeCallback) { + this.resumeCallback(); + this.resumeCallback = null; + } + } + } + + public get isPaused(): boolean { + return this.paused; + } + + public async read(): Promise { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + + if (done) { + return undefined; + } + return value; + } + + public setEncoding(encoding: string): void { + this.encoding = encoding; + } + + public async text(): Promise { + const chunks: ReadFormat[] = []; + + while (true) { + const { done, value } = await this.reader.read(); + if (done) { + break; + } + if (value) { + chunks.push(value); + } + } + + const decoder = new TextDecoder(this.encoding || "utf-8"); + return decoder.decode(await new Blob(chunks).arrayBuffer()); + } + + public async json(): Promise { + const text = await this.text(); + return JSON.parse(text); + } + + private _write(chunk: ReadFormat): void { + this._emit("data", chunk); + } + + private _end(): void { + this._emit("end"); + } + + private _error(error: any): void { + this._emit("error", error); + } + + private _emit(event: string, data?: any): void { + if (this.events[event]) { + for (const callback of this.events[event] || []) { + callback(data); + } + } + } + + private async _startReading(): Promise { + try { + this._emit("readable"); + while (true) { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + this._emit("end"); + this._emit("close"); + break; + } + if (value) { + this._emit("data", value); + } + } + } catch (error) { + this._emit("error", error); + } + } + + [Symbol.asyncIterator](): AsyncIterableIterator { + return { + next: async () => { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + return { done: true, value: undefined }; + } + return { done: false, value }; + }, + [Symbol.asyncIterator]() { + return this; + }, + }; + } +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts new file mode 100644 index 00000000..23c01a1a --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts @@ -0,0 +1,107 @@ +import type { Readable, Writable } from "readable-stream"; + +import { EventCallback, StreamWrapper } from "./chooseStreamWrapper"; + +export class NodePre18StreamWrapper implements StreamWrapper { + private readableStream: Readable; + private encoding: string | undefined; + + constructor(readableStream: Readable) { + this.readableStream = readableStream; + } + + public on(event: string, callback: EventCallback): void { + this.readableStream.on(event, callback); + } + + public off(event: string, callback: EventCallback): void { + this.readableStream.off(event, callback); + } + + public pipe(dest: Writable): Writable { + this.readableStream.pipe(dest); + return dest; + } + + public pipeTo(dest: Writable): Writable { + return this.pipe(dest); + } + + public unpipe(dest?: Writable): void { + if (dest) { + this.readableStream.unpipe(dest); + } else { + this.readableStream.unpipe(); + } + } + + public destroy(error?: Error): void { + this.readableStream.destroy(error); + } + + public pause(): void { + this.readableStream.pause(); + } + + public resume(): void { + this.readableStream.resume(); + } + + public get isPaused(): boolean { + return this.readableStream.isPaused(); + } + + public async read(): Promise { + return new Promise((resolve, reject) => { + const chunk = this.readableStream.read(); + if (chunk) { + resolve(chunk); + } else { + this.readableStream.once("readable", () => { + const chunk = this.readableStream.read(); + resolve(chunk); + }); + this.readableStream.once("error", reject); + } + }); + } + + public setEncoding(encoding?: string): void { + this.readableStream.setEncoding(encoding as BufferEncoding); + this.encoding = encoding; + } + + public async text(): Promise { + const chunks: Uint8Array[] = []; + const encoder = new TextEncoder(); + this.readableStream.setEncoding((this.encoding || "utf-8") as BufferEncoding); + + for await (const chunk of this.readableStream) { + chunks.push(encoder.encode(chunk)); + } + + const decoder = new TextDecoder(this.encoding || "utf-8"); + return decoder.decode(Buffer.concat(chunks)); + } + + public async json(): Promise { + const text = await this.text(); + return JSON.parse(text); + } + + public [Symbol.asyncIterator](): AsyncIterableIterator { + const readableStream = this.readableStream; + const iterator = readableStream[Symbol.asyncIterator](); + + // Create and return an async iterator that yields buffers + return { + async next(): Promise> { + const { value, done } = await iterator.next(); + return { value: value as Buffer, done }; + }, + [Symbol.asyncIterator]() { + return this; + }, + }; + } +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts new file mode 100644 index 00000000..e5e8b680 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts @@ -0,0 +1,244 @@ +import { StreamWrapper } from "./chooseStreamWrapper"; + +type EventCallback = (data?: any) => void; + +export class UndiciStreamWrapper implements StreamWrapper< + UndiciStreamWrapper | WritableStream, + ReadFormat +> { + private readableStream: ReadableStream; + private reader: ReadableStreamDefaultReader; + private events: Record; + private paused: boolean; + private resumeCallback: ((value?: unknown) => void) | null; + private encoding: string | null; + + constructor(readableStream: ReadableStream) { + this.readableStream = readableStream; + this.reader = this.readableStream.getReader(); + this.events = { + data: [], + end: [], + error: [], + readable: [], + close: [], + pause: [], + resume: [], + }; + this.paused = false; + this.resumeCallback = null; + this.encoding = null; + } + + public on(event: string, callback: EventCallback): void { + this.events[event]?.push(callback); + } + + public off(event: string, callback: EventCallback): void { + this.events[event] = this.events[event]?.filter((cb) => cb !== callback); + } + + public pipe( + dest: UndiciStreamWrapper | WritableStream, + ): UndiciStreamWrapper | WritableStream { + this.on("data", (chunk) => { + if (dest instanceof UndiciStreamWrapper) { + dest._write(chunk); + } else { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } + }); + + this.on("end", () => { + if (dest instanceof UndiciStreamWrapper) { + dest._end(); + } else { + const writer = dest.getWriter(); + writer.close(); + } + }); + + this.on("error", (error) => { + if (dest instanceof UndiciStreamWrapper) { + dest._error(error); + } else { + const writer = dest.getWriter(); + writer.abort(error); + } + }); + + this._startReading(); + + return dest; + } + + public pipeTo( + dest: UndiciStreamWrapper | WritableStream, + ): UndiciStreamWrapper | WritableStream { + return this.pipe(dest); + } + + public unpipe(dest: UndiciStreamWrapper | WritableStream): void { + this.off("data", (chunk) => { + if (dest instanceof UndiciStreamWrapper) { + dest._write(chunk); + } else { + const writer = dest.getWriter(); + writer.write(chunk).then(() => writer.releaseLock()); + } + }); + + this.off("end", () => { + if (dest instanceof UndiciStreamWrapper) { + dest._end(); + } else { + const writer = dest.getWriter(); + writer.close(); + } + }); + + this.off("error", (error) => { + if (dest instanceof UndiciStreamWrapper) { + dest._error(error); + } else { + const writer = dest.getWriter(); + writer.abort(error); + } + }); + } + + public destroy(error?: Error): void { + this.reader + .cancel(error) + .then(() => { + this._emit("close"); + }) + .catch((err) => { + this._emit("error", err); + }); + } + + public pause(): void { + this.paused = true; + this._emit("pause"); + } + + public resume(): void { + if (this.paused) { + this.paused = false; + this._emit("resume"); + if (this.resumeCallback) { + this.resumeCallback(); + this.resumeCallback = null; + } + } + } + + public get isPaused(): boolean { + return this.paused; + } + + public async read(): Promise { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + return undefined; + } + return value; + } + + public setEncoding(encoding: string): void { + this.encoding = encoding; + } + + public async text(): Promise { + const chunks: BlobPart[] = []; + + while (true) { + const { done, value } = await this.reader.read(); + if (done) { + break; + } + if (value) { + chunks.push(value); + } + } + + const decoder = new TextDecoder(this.encoding || "utf-8"); + return decoder.decode(await new Blob(chunks).arrayBuffer()); + } + + public async json(): Promise { + const text = await this.text(); + return JSON.parse(text); + } + + private _write(chunk: ReadFormat): void { + this._emit("data", chunk); + } + + private _end(): void { + this._emit("end"); + } + + private _error(error: any): void { + this._emit("error", error); + } + + private _emit(event: string, data?: any): void { + if (this.events[event]) { + for (const callback of this.events[event] || []) { + callback(data); + } + } + } + + private async _startReading(): Promise { + try { + this._emit("readable"); + while (true) { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + this._emit("end"); + this._emit("close"); + break; + } + if (value) { + this._emit("data", value); + } + } + } catch (error) { + this._emit("error", error); + } + } + + [Symbol.asyncIterator](): AsyncIterableIterator { + return { + next: async () => { + if (this.paused) { + await new Promise((resolve) => { + this.resumeCallback = resolve; + }); + } + const { done, value } = await this.reader.read(); + if (done) { + return { done: true, value: undefined }; + } + return { done: false, value }; + }, + [Symbol.asyncIterator]() { + return this; + }, + }; + } +} diff --git a/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/chooseStreamWrapper.ts b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/chooseStreamWrapper.ts new file mode 100644 index 00000000..8c7492fc --- /dev/null +++ b/flowvault/_generated_/rest/src/core/fetcher/stream-wrappers/chooseStreamWrapper.ts @@ -0,0 +1,34 @@ +import type { Readable } from "readable-stream"; + +import { RUNTIME } from "../../runtime"; + +export type EventCallback = (data?: any) => void; + +export interface StreamWrapper { + setEncoding(encoding?: string): void; + on(event: string, callback: EventCallback): void; + off(event: string, callback: EventCallback): void; + pipe(dest: WritableStream): WritableStream; + pipeTo(dest: WritableStream): WritableStream; + unpipe(dest?: WritableStream): void; + destroy(error?: Error): void; + pause(): void; + resume(): void; + get isPaused(): boolean; + read(): Promise; + text(): Promise; + json(): Promise; + [Symbol.asyncIterator](): AsyncIterableIterator; +} + +export async function chooseStreamWrapper(responseBody: any): Promise>> { + if (RUNTIME.type === "node" && RUNTIME.parsedVersion != null && RUNTIME.parsedVersion >= 18) { + return new (await import("./Node18UniversalStreamWrapper")).Node18UniversalStreamWrapper( + responseBody as ReadableStream, + ); + } else if (RUNTIME.type !== "node" && typeof fetch === "function") { + return new (await import("./UndiciStreamWrapper")).UndiciStreamWrapper(responseBody as ReadableStream); + } else { + return new (await import("./NodePre18StreamWrapper")).NodePre18StreamWrapper(responseBody as Readable); + } +} diff --git a/flowvault/_generated_/rest/src/core/index.ts b/flowvault/_generated_/rest/src/core/index.ts new file mode 100644 index 00000000..c752c6a6 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/index.ts @@ -0,0 +1,3 @@ +export * from "./fetcher"; +export * from "./auth"; +export * from "./runtime"; diff --git a/flowvault/_generated_/rest/src/core/json.ts b/flowvault/_generated_/rest/src/core/json.ts new file mode 100644 index 00000000..c052f324 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/json.ts @@ -0,0 +1,27 @@ +/** + * Serialize a value to JSON + * @param value A JavaScript value, usually an object or array, to be converted. + * @param replacer A function that transforms the results. + * @param space Adds indentation, white space, and line break characters to the return-value JSON text to make it easier to read. + * @returns JSON string + */ +export const toJson = ( + value: unknown, + replacer?: (this: unknown, key: string, value: unknown) => unknown, + space?: string | number, +): string => { + return JSON.stringify(value, replacer, space); +}; + +/** + * Parse JSON string to object, array, or other type + * @param text A valid JSON string. + * @param reviver A function that transforms the results. This function is called for each member of the object. If a member contains nested objects, the nested objects are transformed before the parent object is. + * @returns Parsed object, array, or other type + */ +export function fromJson( + text: string, + reviver?: (this: unknown, key: string, value: unknown) => unknown, +): T { + return JSON.parse(text, reviver); +} diff --git a/flowvault/_generated_/rest/src/core/runtime/index.ts b/flowvault/_generated_/rest/src/core/runtime/index.ts new file mode 100644 index 00000000..5c76dbb1 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/runtime/index.ts @@ -0,0 +1 @@ +export { RUNTIME } from "./runtime"; diff --git a/flowvault/_generated_/rest/src/core/runtime/runtime.ts b/flowvault/_generated_/rest/src/core/runtime/runtime.ts new file mode 100644 index 00000000..a9750175 --- /dev/null +++ b/flowvault/_generated_/rest/src/core/runtime/runtime.ts @@ -0,0 +1,131 @@ +interface DenoGlobal { + version: { + deno: string; + }; +} + +interface BunGlobal { + version: string; +} + +declare const Deno: DenoGlobal | undefined; +declare const Bun: BunGlobal | undefined; +declare const EdgeRuntime: string | undefined; + +/** + * A constant that indicates which environment and version the SDK is running in. + */ +export const RUNTIME: Runtime = evaluateRuntime(); + +export interface Runtime { + type: "browser" | "web-worker" | "deno" | "bun" | "node" | "react-native" | "unknown" | "workerd" | "edge-runtime"; + version?: string; + parsedVersion?: number; +} + +function evaluateRuntime(): Runtime { + /** + * A constant that indicates whether the environment the code is running is a Web Browser. + */ + const isBrowser = typeof window !== "undefined" && typeof window.document !== "undefined"; + if (isBrowser) { + return { + type: "browser", + version: window.navigator.userAgent, + }; + } + + /** + * A constant that indicates whether the environment the code is running is Cloudflare. + * https://developers.cloudflare.com/workers/runtime-apis/web-standards/#navigatoruseragent + */ + const isCloudflare = typeof globalThis !== "undefined" && globalThis?.navigator?.userAgent === "Cloudflare-Workers"; + if (isCloudflare) { + return { + type: "workerd", + }; + } + + /** + * A constant that indicates whether the environment the code is running is Edge Runtime. + * https://vercel.com/docs/functions/runtimes/edge-runtime#check-if-you're-running-on-the-edge-runtime + */ + const isEdgeRuntime = typeof EdgeRuntime === "string"; + if (isEdgeRuntime) { + return { + type: "edge-runtime", + }; + } + + /** + * A constant that indicates whether the environment the code is running is a Web Worker. + */ + const isWebWorker = + typeof self === "object" && + // @ts-ignore + typeof self?.importScripts === "function" && + (self.constructor?.name === "DedicatedWorkerGlobalScope" || + self.constructor?.name === "ServiceWorkerGlobalScope" || + self.constructor?.name === "SharedWorkerGlobalScope"); + if (isWebWorker) { + return { + type: "web-worker", + }; + } + + /** + * A constant that indicates whether the environment the code is running is Deno. + * FYI Deno spoofs process.versions.node, see https://deno.land/std@0.177.0/node/process.ts?s=versions + */ + const isDeno = + typeof Deno !== "undefined" && typeof Deno.version !== "undefined" && typeof Deno.version.deno !== "undefined"; + if (isDeno) { + return { + type: "deno", + version: Deno.version.deno, + }; + } + + /** + * A constant that indicates whether the environment the code is running is Bun.sh. + */ + const isBun = typeof Bun !== "undefined" && typeof Bun.version !== "undefined"; + if (isBun) { + return { + type: "bun", + version: Bun.version, + }; + } + + /** + * A constant that indicates whether the environment the code is running is Node.JS. + */ + const isNode = + typeof process !== "undefined" && + "version" in process && + !!process.version && + "versions" in process && + !!process.versions?.node; + if (isNode) { + return { + type: "node", + version: process.versions.node, + parsedVersion: Number(process.versions.node.split(".")[0]), + }; + } + + /** + * A constant that indicates whether the environment the code is running is in React-Native. + * https://github.com/facebook/react-native/blob/main/packages/react-native/Libraries/Core/setUpNavigator.js + */ + const isReactNative = typeof navigator !== "undefined" && navigator?.product === "ReactNative"; + if (isReactNative) { + return { + type: "react-native", + }; + } + + return { + type: "unknown", + }; +} diff --git a/flowvault/_generated_/rest/src/environments.ts b/flowvault/_generated_/rest/src/environments.ts new file mode 100644 index 00000000..c5f61083 --- /dev/null +++ b/flowvault/_generated_/rest/src/environments.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export const SkyflowEnvironment = { + Production: "https://%7B%7Bvault_url%7D%7D", + Sandbox: "https://%7B%7Bvault_url%7D%7D", +} as const; + +export type SkyflowEnvironment = typeof SkyflowEnvironment.Production | typeof SkyflowEnvironment.Sandbox; diff --git a/flowvault/_generated_/rest/src/errors/SkyflowError.ts b/flowvault/_generated_/rest/src/errors/SkyflowError.ts new file mode 100644 index 00000000..003768c2 --- /dev/null +++ b/flowvault/_generated_/rest/src/errors/SkyflowError.ts @@ -0,0 +1,55 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as core from "../core"; +import { toJson } from "../core/json"; + +export class SkyflowError extends Error { + public readonly statusCode?: number; + public readonly body?: unknown; + public readonly rawResponse?: core.RawResponse; + + constructor({ + message, + statusCode, + body, + rawResponse, + }: { + message?: string; + statusCode?: number; + body?: unknown; + rawResponse?: core.RawResponse; + }) { + super(buildMessage({ message, statusCode, body })); + Object.setPrototypeOf(this, SkyflowError.prototype); + this.statusCode = statusCode; + this.body = body; + this.rawResponse = rawResponse; + } +} + +function buildMessage({ + message, + statusCode, + body, +}: { + message: string | undefined; + statusCode: number | undefined; + body: unknown | undefined; +}): string { + let lines: string[] = []; + if (message != null) { + lines.push(message); + } + + if (statusCode != null) { + lines.push(`Status code: ${statusCode.toString()}`); + } + + if (body != null) { + lines.push(`Body: ${toJson(body, undefined, 2)}`); + } + + return lines.join("\n"); +} diff --git a/flowvault/_generated_/rest/src/errors/SkyflowTimeoutError.ts b/flowvault/_generated_/rest/src/errors/SkyflowTimeoutError.ts new file mode 100644 index 00000000..75024f63 --- /dev/null +++ b/flowvault/_generated_/rest/src/errors/SkyflowTimeoutError.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export class SkyflowTimeoutError extends Error { + constructor(message: string) { + super(message); + Object.setPrototypeOf(this, SkyflowTimeoutError.prototype); + } +} diff --git a/flowvault/_generated_/rest/src/errors/index.ts b/flowvault/_generated_/rest/src/errors/index.ts new file mode 100644 index 00000000..5567f00d --- /dev/null +++ b/flowvault/_generated_/rest/src/errors/index.ts @@ -0,0 +1,2 @@ +export { SkyflowError } from "./SkyflowError"; +export { SkyflowTimeoutError } from "./SkyflowTimeoutError"; diff --git a/flowvault/_generated_/rest/src/index.ts b/flowvault/_generated_/rest/src/index.ts new file mode 100644 index 00000000..0706e084 --- /dev/null +++ b/flowvault/_generated_/rest/src/index.ts @@ -0,0 +1,4 @@ +export * as Skyflow from "./api"; +export { SkyflowClient } from "./Client"; +export { SkyflowEnvironment } from "./environments"; +export { SkyflowError, SkyflowTimeoutError } from "./errors"; diff --git a/flowvault/_generated_/rest/src/version.ts b/flowvault/_generated_/rest/src/version.ts new file mode 100644 index 00000000..e3655fb3 --- /dev/null +++ b/flowvault/_generated_/rest/src/version.ts @@ -0,0 +1 @@ +export const SDK_VERSION = "1.0.21"; diff --git a/flowvault/api-extractor.json b/flowvault/api-extractor.json new file mode 100644 index 00000000..0f7db7c9 --- /dev/null +++ b/flowvault/api-extractor.json @@ -0,0 +1,32 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json", + "projectFolder": ".", + "mainEntryPointFilePath": "/dist/index.d.ts", + "bundledPackages": [], + "compiler": { + "tsconfigFilePath": "/tsconfig.dts.json" + }, + "apiReport": { + "enabled": true, + "reportFolder": "/api-report/", + "reportTempFolder": "/temp/", + "reportFileName": ".api.md" + }, + "docModel": { "enabled": false }, + "dtsRollup": { "enabled": false }, + "tsdocMetadata": { "enabled": false }, + "messages": { + "compilerMessageReporting": { + "default": { "logLevel": "warning" } + }, + "extractorMessageReporting": { + "default": { "logLevel": "warning" }, + "ae-missing-release-tag": { "logLevel": "none" }, + "ae-setter-with-docs": { "logLevel": "none" }, + "ae-forgotten-export": { "logLevel": "none" } + }, + "tsdocMessageReporting": { + "default": { "logLevel": "none" } + } + } +} diff --git a/flowvault/api-report/skyflow-flowvault-node.api.md b/flowvault/api-report/skyflow-flowvault-node.api.md new file mode 100644 index 00000000..b97f01fc --- /dev/null +++ b/flowvault/api-report/skyflow-flowvault-node.api.md @@ -0,0 +1,618 @@ +## API Report File for "skyflow-flowvault-node" + +> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/). + +```ts + +import { AuthInfo } from 'skyflow-common'; +import { BaseDeleteFilesRequest } from 'skyflow-common'; +import { BaseDeleteFilesResponse } from 'skyflow-common'; +import { BaseDeleteRequest } from 'skyflow-common'; +import { BaseDeleteResponse } from 'skyflow-common'; +import { BaseDetokenizeRequest } from 'skyflow-common'; +import { BaseDetokenizeResponse } from 'skyflow-common'; +import { BaseGetRequest } from 'skyflow-common'; +import { BaseGetResponse } from 'skyflow-common'; +import { BaseGetTokensRequest } from 'skyflow-common'; +import { BaseGetTokensResponse } from 'skyflow-common'; +import { BaseInsertRequest } from 'skyflow-common'; +import { BaseInsertResponse } from 'skyflow-common'; +import { BaseQueryRequest } from 'skyflow-common'; +import { BaseQueryResponse } from 'skyflow-common'; +import { BaseSkyflow } from 'skyflow-common'; +import { BaseUpdateRequest } from 'skyflow-common'; +import { BaseUpdateResponse } from 'skyflow-common'; +import { BaseUploadFilesRequest } from 'skyflow-common'; +import { BaseUploadFilesResponse } from 'skyflow-common'; +import { BaseVaultEntry } from 'skyflow-common'; +import { Credentials } from 'skyflow-common'; +import { LogLevel } from 'skyflow-common'; +import { SkyflowConfig } from 'skyflow-common'; +import { SkyflowError } from 'skyflow-common'; +import { SkyflowRecordError } from 'skyflow-common'; +import { VaultConfig } from 'skyflow-common'; + +// @public +export interface ColumnRedaction { + // (undocumented) + columnName: string; + // (undocumented) + redaction: string; +} + +// @public +export enum CustomHeaderKey { + // (undocumented) + REQUEST_ID_HEADER = "x-request-id", + // (undocumented) + SKYFLOW_ACCOUNT_ID = "x-skyflow-account-id", + // (undocumented) + SKYFLOW_ACCOUNT_NAME = "x-skyflow-account-name" +} + +// @public +export class DeleteFilesOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export interface DeleteFilesRecord { + // (undocumented) + columns: string[]; + // (undocumented) + skyflowId?: string; + // (undocumented) + tableName: string; + // (undocumented) + uniqueValues?: Record[]; +} + +// @public +export class DeleteFilesRequest extends BaseDeleteFilesRequest { + constructor(records: DeleteFilesRecord[]); + // (undocumented) + get records(): DeleteFilesRecord[]; + set records(value: DeleteFilesRecord[]); +} + +// @public +export class DeleteFilesResponse extends BaseDeleteFilesResponse { + constructor(input: { + records: DeleteFilesResponseRecord[]; + }); + // (undocumented) + readonly records: DeleteFilesResponseRecord[]; +} + +// @public (undocumented) +export interface DeleteFilesResponseColumn { + // (undocumented) + column: string; + // (undocumented) + status: string | null; +} + +// @public (undocumented) +export interface DeleteFilesResponseRecord { + // (undocumented) + columns: DeleteFilesResponseColumn[] | null; + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; + // (undocumented) + tableName?: string | null; +} + +// @public +export class DeleteOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export class DeleteRequest extends BaseDeleteRequest { + constructor(table: string, ids: string[], uniqueValues?: Record[]); + // (undocumented) + get ids(): string[]; + set ids(value: string[]); + // (undocumented) + get table(): string; + set table(value: string); + // (undocumented) + get uniqueValues(): Record[] | undefined; + set uniqueValues(value: Record[] | undefined); +} + +// @public +export class DeleteResponse extends BaseDeleteResponse { + constructor(input: { + records: DeleteResponseRecord[]; + }); + // (undocumented) + readonly records: DeleteResponseRecord[]; +} + +// @public (undocumented) +export interface DeleteResponseRecord { + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; +} + +// @public +export class DetokenizeOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export class DetokenizeRequest extends BaseDetokenizeRequest { + constructor(tokens: string[], tokenGroupRedactions?: TokenGroupRedaction[]); + // (undocumented) + get tokenGroupRedactions(): TokenGroupRedaction[] | undefined; + set tokenGroupRedactions(value: TokenGroupRedaction[] | undefined); + // (undocumented) + get tokens(): string[]; + set tokens(value: string[]); +} + +// @public +export class DetokenizeResponse extends BaseDetokenizeResponse { + constructor(input: { + records: DetokenizeResponseRecord[]; + }); + // (undocumented) + readonly records: DetokenizeResponseRecord[]; +} + +// @public (undocumented) +export interface DetokenizeResponseRecord { + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + metadata?: { + skyflowId?: string | null; + tableName?: string | null; + } | null; + // (undocumented) + requestId?: string | null; + // (undocumented) + token: string; + // (undocumented) + tokenGroupName?: string | null; + // (undocumented) + value?: unknown; +} + +// @public +export class FlowVaultOptions { + getInterceptor(): RequestInterceptor | undefined; + setInterceptor(interceptor: RequestInterceptor): void; +} + +// @public +export type FlowVaultRecordError = SkyflowRecordError; + +// @public +export class GetOptions extends FlowVaultOptions { + constructor(); + // (undocumented) + getColumnRedactions(): ColumnRedaction[] | undefined; + // (undocumented) + getFields(): string[] | undefined; + // (undocumented) + getLimit(): number | undefined; + // (undocumented) + getOffset(): number | undefined; + // (undocumented) + getUniqueValues(): Record[] | undefined; + // (undocumented) + setColumnRedactions(columnRedactions: ColumnRedaction[]): void; + // (undocumented) + setFields(fields: string[]): void; + // (undocumented) + setLimit(limit: number): void; + // (undocumented) + setOffset(offset: number): void; + // (undocumented) + setUniqueValues(uniqueValues: Record[]): void; +} + +// @public +export interface GetRecord { + // (undocumented) + columnRedactions?: ColumnRedaction[]; + // (undocumented) + columns?: string[]; + // (undocumented) + ids: string[]; + // (undocumented) + tableName: string; + // (undocumented) + uniqueValues?: Record[]; +} + +// @public +export class GetRequest extends BaseGetRequest { + constructor(table: string, ids: string[], records?: GetRecord[]); + // (undocumented) + get ids(): string[]; + set ids(value: string[]); + // (undocumented) + get records(): GetRecord[] | undefined; + set records(value: GetRecord[] | undefined); + // (undocumented) + get table(): string; + set table(value: string); +} + +// @public +export class GetResponse extends BaseGetResponse { + constructor(input: { + records: GetResponseRecord[]; + }); + // (undocumented) + readonly records: GetResponseRecord[]; +} + +// @public (undocumented) +export interface GetResponseRecord { + // (undocumented) + data?: Record | null; + // (undocumented) + error?: string | null; + // (undocumented) + hashedData?: Record | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; + // (undocumented) + tableName?: string | null; + // (undocumented) + tokens?: Record | null; +} + +// @public +export class GetTokensOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export interface GetTokensRecord { + // (undocumented) + tokenGroupName: string; + // (undocumented) + value: unknown; +} + +// @public +export class GetTokensRequest extends BaseGetTokensRequest { + constructor(records: GetTokensRecord[]); + // (undocumented) + get records(): GetTokensRecord[]; + set records(value: GetTokensRecord[]); +} + +// @public +export class GetTokensResponse extends BaseGetTokensResponse { + constructor(input: { + records: GetTokensResponseRecord[]; + }); + // (undocumented) + readonly records: GetTokensResponseRecord[]; +} + +// @public (undocumented) +export interface GetTokensResponseRecord { + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + token?: string | null; + // (undocumented) + tokenGroupName: string | null; + // (undocumented) + value: unknown; +} + +// @public +export class InsertOptions extends FlowVaultOptions { + constructor(); + // (undocumented) + getUpsert(): UpsertOptions | undefined; + // (undocumented) + setUpsert(upsert: UpsertOptions): void; +} + +// @public +export interface InsertRecord { + // (undocumented) + data: Record; + // (undocumented) + table?: string; + // (undocumented) + tokens?: Record; + upsert?: UpsertOptions; +} + +// @public +export class InsertRequest extends BaseInsertRequest { + constructor(tableName: string, records: InsertRecord[]); + // (undocumented) + get records(): InsertRecord[]; + set records(value: InsertRecord[]); + // (undocumented) + get tableName(): string; + set tableName(value: string); +} + +// @public +export class InsertResponse extends BaseInsertResponse { + constructor(input: { + records: InsertResponseRecord[]; + }); + // (undocumented) + readonly records: InsertResponseRecord[]; +} + +// @public (undocumented) +export interface InsertResponseRecord { + // (undocumented) + data?: Record | null; + // (undocumented) + error?: string | null; + // (undocumented) + hashedData?: Record | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; + // (undocumented) + tableName?: string | null; + // (undocumented) + tokens?: Record | null; +} + +// @public +export class QueryOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export class QueryRequest extends BaseQueryRequest { + constructor(query: string); + // (undocumented) + get query(): string; + set query(value: string); +} + +// @public +export class QueryResponse extends BaseQueryResponse { + constructor(input: { + fields: Record[]; + errors: FlowVaultRecordError[] | null; + metadata: QueryResponseMetadata; + requestId: string; + }); + // (undocumented) + readonly errors: FlowVaultRecordError[] | null; + // (undocumented) + readonly fields: Record[]; + // (undocumented) + readonly metadata: QueryResponseMetadata; + // (undocumented) + readonly requestId: string; +} + +// @public +export interface QueryResponseMetadata { + // (undocumented) + columns: string[]; +} + +// @public +export class RequestContext { + constructor(operation: string, batchIndex?: number, totalBatches?: number); + addHeader(key: CustomHeaderKey, value: string): void; + readonly batchIndex: number; + get headers(): Map; + // (undocumented) + readonly operation: string; + readonly totalBatches: number; +} + +// @public +export type RequestInterceptor = (context: RequestContext) => void; + +// Warning: (ae-forgotten-export) The symbol "ClientEntry" needs to be exported by the entry point index.d.ts +// +// @public +class Skyflow extends BaseSkyflow { + constructor(config: SkyflowConfig); + // (undocumented) + protected buildVaultEntry(config: VaultConfig): ClientEntry; + // Warning: (ae-forgotten-export) The symbol "VaultController" needs to be exported by the entry point index.d.ts + // + // (undocumented) + vault(vaultId?: string): VaultController; +} +export { Skyflow } +export default Skyflow; + +// @public +export interface TokenGroupRedaction { + // (undocumented) + redaction: string; + // (undocumented) + tokenGroupName: string; +} + +// @public +export class UpdateOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export interface UpdateRecord { + // (undocumented) + data: Record; + // (undocumented) + skyflowId: string; + // (undocumented) + table?: string; + // (undocumented) + tokens?: Record; +} + +// @public +export class UpdateRequest extends BaseUpdateRequest { + constructor(tableName: string, records: UpdateRecord[], updateType?: string); + // (undocumented) + get records(): UpdateRecord[]; + set records(value: UpdateRecord[]); + // (undocumented) + get tableName(): string; + set tableName(value: string); + // (undocumented) + get updateType(): string | undefined; + set updateType(value: string | undefined); +} + +// @public +export class UpdateResponse extends BaseUpdateResponse { + constructor(input: { + records: UpdateResponseRecord[]; + }); + // (undocumented) + readonly records: UpdateResponseRecord[]; +} + +// @public (undocumented) +export interface UpdateResponseRecord { + // (undocumented) + data?: Record | null; + // (undocumented) + error?: string | null; + // (undocumented) + hashedData?: Record | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; + // (undocumented) + tableName?: string | null; + // (undocumented) + tokens?: Record | null; +} + +// @public +export interface UploadFilesColumn { + // (undocumented) + base64?: string; + // (undocumented) + column: string; + // (undocumented) + file?: File; + // (undocumented) + fileName?: string; + // (undocumented) + filePath?: string; +} + +// @public +export class UploadFilesOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export interface UploadFilesRecord { + // (undocumented) + columns: UploadFilesColumn[]; + // (undocumented) + skyflowId?: string; + // (undocumented) + tableName: string; +} + +// @public +export class UploadFilesRequest extends BaseUploadFilesRequest { + constructor(records: UploadFilesRecord[]); + // (undocumented) + get records(): UploadFilesRecord[]; + set records(value: UploadFilesRecord[]); +} + +// @public +export class UploadFilesResponse extends BaseUploadFilesResponse { + constructor(input: { + records: UploadFilesResponseRecord[]; + }); + // (undocumented) + readonly records: UploadFilesResponseRecord[]; +} + +// @public (undocumented) +export interface UploadFilesResponseColumn { + // (undocumented) + column: string; + // (undocumented) + error?: string | null; + // (undocumented) + fileName?: string | null; + // (undocumented) + uploadStatus: UploadStatus; +} + +// @public (undocumented) +export interface UploadFilesResponseRecord { + // (undocumented) + columns: UploadFilesResponseColumn[]; + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; + // (undocumented) + tableName?: string | null; +} + +// @public +export type UploadStatus = 'UPLOADED' | 'FAILED' | 'SKIPPED'; + +// @public +export interface UpsertOptions { + // (undocumented) + uniqueColumns: string[]; + // (undocumented) + updateType: string; +} + + +export * from "skyflow-common"; + +// (No @packageDocumentation comment for this package) + +``` diff --git a/flowvault/jest.config.js b/flowvault/jest.config.js new file mode 100644 index 00000000..b47c51fb --- /dev/null +++ b/flowvault/jest.config.js @@ -0,0 +1,19 @@ +module.exports = { + rootDir: '.', + testEnvironment: 'node', + testMatch: ['/test/**/*.test.ts'], + transform: { + '^.+\\.ts$': [ + 'ts-jest', + { + tsconfig: '/tsconfig.test.json', + }, + ], + }, + moduleFileExtensions: ['ts', 'js', 'json'], + // Coverage reflects the hand-written layer only; the generated (maintainer-owned) + // client is excluded (mirrors skyvault's jest config). + collectCoverageFrom: ['src/**/*.ts'], + coveragePathIgnorePatterns: ['/node_modules/', '/_generated_/'], + verbose: true, +}; diff --git a/flowvault/package.json b/flowvault/package.json new file mode 100644 index 00000000..ac3d372a --- /dev/null +++ b/flowvault/package.json @@ -0,0 +1,61 @@ +{ + "name": "skyflow-flowvault-node", + "version": "1.0.0", + "description": "Skyflow FlowVault SDK for Node.js", + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "require": "./dist/index.js", + "default": "./dist/index.js" + } + }, + "files": [ + "dist", + "README.md", + "LICENSE" + ], + "scripts": { + "build": "tsc -b", + "build:dist": "tsup", + "test": "jest --coverage", + "contract-snapshot": "api-extractor run --local", + "contract-snapshot-verify": "api-extractor run" + }, + "repository": { + "type": "git", + "url": "https://github.com/skyflowapi/skyflow-node-sdk.git" + }, + "browser": { + "child_process": false + }, + "author": "", + "license": "MIT", + "keywords": [ + "skyflow", + "privacy", + "api", + "security", + "vault", + "flowvault", + "PII", + "tokenization" + ], + "dependencies": { + "dotenv": "16.6.1", + "form-data": "4.0.6", + "form-data-encoder": "4.1.0", + "formdata-node": "6.0.3", + "js-base64": "3.7.7", + "jsonwebtoken": "9.0.3", + "jwt-decode": "4.0.0", + "node-fetch": "2.7.0", + "qs": "6.15.2", + "readable-stream": "4.7.0", + "url-join": "4.0.1" + }, + "devDependencies": { + "skyflow-common": "*" + } +} diff --git a/flowvault/samples/README.md b/flowvault/samples/README.md new file mode 100644 index 00000000..685930dd --- /dev/null +++ b/flowvault/samples/README.md @@ -0,0 +1,66 @@ +# FlowVault Node samples + +Runnable examples for the `skyflow-flowvault-node` SDK — one file per operation. See the +[flowvault README](../README.md) for the full SDK guide. + +## Prerequisites + +- Node.js 20+ (the `File` API used by `uploadFiles` requires v20+). +- `npm install skyflow-flowvault-node` +- A FlowVault vault and Skyflow credentials (a service-account `credentials.json`, an API key, or a + bearer token). + +## Configure + +Each sample has placeholders near the top — replace them with your own values: + +```typescript +const credentials: Credentials = { path: '' }; // or { apiKey } / { token } / { credentialsString } + +const skyflowClient = new Skyflow({ + vaultConfigs: [{ + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }], + logLevel: LogLevel.ERROR, +}); +``` + +## Run + +The samples are TypeScript. Run one with `ts-node` (or compile with `tsc` first): + +```bash +npx ts-node flowvault/samples/vault-api/insert-records.ts +``` + +## Vault operations + +| Sample | Demonstrates | +|---|---| +| [insert-records.ts](vault-api/insert-records.ts) | Insert records (request-level or per-record `table`; `upsert`) | +| [get-records.ts](vault-api/get-records.ts) | Retrieve records by Skyflow ID (single- and multi-table) | +| [update-record.ts](vault-api/update-record.ts) | Update a record by Skyflow ID | +| [delete-records.ts](vault-api/delete-records.ts) | Delete records by Skyflow ID or unique values | +| [detokenize-records.ts](vault-api/detokenize-records.ts) | Detokenize tokens | +| [query-records.ts](vault-api/query-records.ts) | Run a read-only SQL `SELECT` query | +| [get-tokens.ts](vault-api/get-tokens.ts) | Look up existing tokens for values (deterministic token groups) | +| [upload-files.ts](vault-api/upload-files.ts) | Upload files to file columns (`filePath` / `base64` / `file`) | +| [delete-files.ts](vault-api/delete-files.ts) | Delete files from file columns (by `skyflowId` or `uniqueValues`) | + +## Custom headers & HTTP config + +| Sample | Demonstrates | +|---|---| +| [custom-header-example.ts](vault-api/custom-header-example.ts) | Attach supported custom headers via `options.setInterceptor(...)` and `CustomHeaderKey` | +| [timeout-and-retry-config-example.ts](vault-api/timeout-and-retry-config-example.ts) | Per-vault `timeout` / `maxRetries` / `vaultUrl` config | + +## Service account (token generation) + +| Sample | Demonstrates | +|---|---| +| [token-generation-example.ts](service-account/token-generation-example.ts) | Bearer token from a credentials file path or a credentials string | +| [scoped-token-generation-example.ts](service-account/scoped-token-generation-example.ts) | Scoped tokens via `roleIds` | +| [signed-token-generation-example.ts](service-account/signed-token-generation-example.ts) | Signed data tokens (`generateSignedDataTokens`), string and object context | diff --git a/flowvault/samples/service-account/scoped-token-generation-example.ts b/flowvault/samples/service-account/scoped-token-generation-example.ts new file mode 100644 index 00000000..c766e42d --- /dev/null +++ b/flowvault/samples/service-account/scoped-token-generation-example.ts @@ -0,0 +1,46 @@ +import { + generateBearerToken, + generateBearerTokenFromCreds, + isExpired, + LogLevel, +} from 'skyflow-flowvault-node'; + +const filepath = ''; +let bearerToken = ''; + +const cred = { + clientID: '', + clientName: '', + keyID: '', + tokenURI: '', + privateKey: '', +}; + +// Scoped tokens restrict access to specific roles via roleIds. +async function getScopedBearerTokenFromFilePath(): Promise { + const options = { + roleIds: ['', ''], + // tokenUri: '', // optional + logLevel: LogLevel.WARN, + }; + if (!isExpired(bearerToken)) return bearerToken; + const response = await generateBearerToken(filepath, options); + bearerToken = response.accessToken; + return bearerToken; +} + +async function getScopedBearerTokenFromCreds(): Promise { + const options = { + roleIds: ['', ''], + logLevel: LogLevel.WARN, + }; + if (!isExpired(bearerToken)) return bearerToken; + const response = await generateBearerTokenFromCreds(JSON.stringify(cred), options); + bearerToken = response.accessToken; + return bearerToken; +} + +(async () => { + console.log('Scoped Token (from file):', await getScopedBearerTokenFromFilePath()); + console.log('Scoped Token (from string):', await getScopedBearerTokenFromCreds()); +})(); diff --git a/flowvault/samples/service-account/signed-token-generation-example.ts b/flowvault/samples/service-account/signed-token-generation-example.ts new file mode 100644 index 00000000..3719fc42 --- /dev/null +++ b/flowvault/samples/service-account/signed-token-generation-example.ts @@ -0,0 +1,52 @@ +import { + generateSignedDataTokens, + generateSignedDataTokensFromCreds, +} from 'skyflow-flowvault-node'; + +const filepath = ''; + +const cred = { + clientID: '', + clientName: '', + keyID: '', + tokenURI: '', + privateKey: '', +}; + +// Approach 1: signed data tokens with a string context. +async function getSignedTokensWithStringContext() { + const options = { + ctx: 'user_12345', + dataTokens: ['', ''], + timeToLive: 30, // in seconds + }; + return generateSignedDataTokens(filepath, options); +} + +// Approach 2: signed data tokens with an object context. Each key maps to a +// Skyflow CEL policy variable under request.context.*, e.g. +// request.context.role == "analyst" && request.context.department == "research" +async function getSignedTokensWithObjectContext() { + const options = { + ctx: { role: 'analyst', department: 'research', user_id: 'user_67890' }, + dataTokens: ['', ''], + timeToLive: 30, + }; + return generateSignedDataTokens(filepath, options); +} + +// Approach 3: signed data tokens from a credentials string. +async function getSignedTokensFromCreds() { + const options = { + ctx: 'user_12345', + dataTokens: ['', ''], + timeToLive: 30, + }; + return generateSignedDataTokensFromCreds(JSON.stringify(cred), options); +} + +(async () => { + console.log('String context:', await getSignedTokensWithStringContext()); + console.log('Object context:', await getSignedTokensWithObjectContext()); + console.log('Creds string:', await getSignedTokensFromCreds()); +})(); diff --git a/flowvault/samples/service-account/token-generation-example.ts b/flowvault/samples/service-account/token-generation-example.ts new file mode 100644 index 00000000..a50642da --- /dev/null +++ b/flowvault/samples/service-account/token-generation-example.ts @@ -0,0 +1,38 @@ +import { + generateBearerToken, + generateBearerTokenFromCreds, + isExpired, +} from 'skyflow-flowvault-node'; + +const filepath = ''; +let bearerToken = ''; + +// To generate a Bearer Token from a credentials string. +const cred = { + clientID: '', + clientName: '', + keyID: '', + tokenURI: '', + privateKey: '', +}; + +// Generate a Bearer Token from a credentials file path. Reuse it until it expires. +async function getBearerTokenFromFilePath(): Promise { + if (!isExpired(bearerToken)) return bearerToken; + const response = await generateBearerToken(filepath); + bearerToken = response.accessToken; + return bearerToken; +} + +// Generate a Bearer Token from a credentials string. +async function getBearerTokenFromCreds(): Promise { + if (!isExpired(bearerToken)) return bearerToken; + const response = await generateBearerTokenFromCreds(JSON.stringify(cred)); + bearerToken = response.accessToken; + return bearerToken; +} + +(async () => { + console.log('Generated Bearer Token (from file):', await getBearerTokenFromFilePath()); + console.log('Generated Bearer Token (from string):', await getBearerTokenFromCreds()); +})(); diff --git a/flowvault/samples/vault-api/custom-header-example.ts b/flowvault/samples/vault-api/custom-header-example.ts new file mode 100644 index 00000000..6157b6f1 --- /dev/null +++ b/flowvault/samples/vault-api/custom-header-example.ts @@ -0,0 +1,69 @@ +import { randomUUID } from 'crypto'; +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + InsertRequest, + InsertOptions, + CustomHeaderKey, + Credentials, +} from 'skyflow-flowvault-node'; + +/** + * Attach custom request headers to a call. The SDK supports a fixed set of + * headers (see CustomHeaderKey): SKYFLOW_ACCOUNT_ID (x-skyflow-account-id), + * SKYFLOW_ACCOUNT_NAME (x-skyflow-account-name) and REQUEST_ID_HEADER + * (x-request-id). Register an interceptor on the *Options object; it runs once + * per request and receives a RequestContext whose `operation` names the call. + */ +async function performInsertWithCustomHeaders(): Promise { + try { + const credentials: Credentials = { + token: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.DEV, + credentials, + }, + ], + logLevel: LogLevel.DEBUG, + }); + + const insertRequest = new InsertRequest('', [ + { data: { '': '' } }, + ]); + + const requestId = randomUUID(); + console.log('request id =>', requestId); + + const insertOptions = new InsertOptions(); + insertOptions.setInterceptor((ctx) => { + // pass the request id (or any supported CustomHeaderKey) here + ctx.addHeader(CustomHeaderKey.REQUEST_ID_HEADER, requestId); + }); + + const response = await skyflowClient + .vault('') + .insert(insertRequest, insertOptions); + + console.log('Records:', response.records); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performInsertWithCustomHeaders(); diff --git a/flowvault/samples/vault-api/delete-files.ts b/flowvault/samples/vault-api/delete-files.ts new file mode 100644 index 00000000..758d1786 --- /dev/null +++ b/flowvault/samples/vault-api/delete-files.ts @@ -0,0 +1,60 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + DeleteFilesRequest, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Delete files from file columns (by Skyflow ID or by unique values). */ +async function performDeleteFiles(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // Each record sets exactly ONE of skyflowId or uniqueValues to target the record(s). + const deleteFilesRequest = new DeleteFilesRequest([ + { + tableName: '', + columns: [''], + skyflowId: '', + // uniqueValues: [{ email: 'john@example.com' }], + }, + ]); + + const response = await skyflowClient + .vault('') + .deleteFiles(deleteFilesRequest); + + // response.records: one DeleteFilesResponseRecord per resolved record, in order. + // Each has skyflowId, tableName, columns, httpCode, error, requestId. Each column + // result has column and status ("DELETED"); columns is null on a failed record. + console.log('Records:', JSON.stringify(response.records, null, 2)); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performDeleteFiles(); diff --git a/flowvault/samples/vault-api/delete-records.ts b/flowvault/samples/vault-api/delete-records.ts new file mode 100644 index 00000000..0f517654 --- /dev/null +++ b/flowvault/samples/vault-api/delete-records.ts @@ -0,0 +1,57 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + DeleteRequest, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Delete records by Skyflow ID or by unique column values. */ +async function performSecureDataDeletion(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // Target by Skyflow IDs... + const deleteRequest = new DeleteRequest('', ['', '']); + + // ...or by unique values: a list of { column: value } maps. + // const deleteRequest = new DeleteRequest('', [], [ + // { email: 'john@example.com' }, + // ]); + + const response = await skyflowClient + .vault('') + .delete(deleteRequest); + + // response.records: one DeleteResponseRecord per input, in order. Each has + // skyflowId, httpCode, error, requestId. + console.log('Records:', response.records); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performSecureDataDeletion(); diff --git a/flowvault/samples/vault-api/detokenize-records.ts b/flowvault/samples/vault-api/detokenize-records.ts new file mode 100644 index 00000000..57e6350b --- /dev/null +++ b/flowvault/samples/vault-api/detokenize-records.ts @@ -0,0 +1,56 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + DetokenizeRequest, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Detokenize a list of tokens back to their values. */ +async function performSecureDetokenization(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // Optionally pass group-level redactions; omit for the default redaction. + const detokenizeRequest = new DetokenizeRequest( + ['', ''], + // [{ tokenGroupName: '', redaction: 'PLAIN_TEXT' }], + ); + + const response = await skyflowClient + .vault('') + .detokenize(detokenizeRequest); + + // response.records: one DetokenizeResponseRecord per input, in order. Each has + // token, value, tokenGroupName, metadata ({ skyflowId, tableName }), + // httpCode, error, requestId. + console.log('Records:', response.records); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performSecureDetokenization(); diff --git a/flowvault/samples/vault-api/get-records.ts b/flowvault/samples/vault-api/get-records.ts new file mode 100644 index 00000000..5cd8c46c --- /dev/null +++ b/flowvault/samples/vault-api/get-records.ts @@ -0,0 +1,66 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + GetRequest, + GetOptions, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Retrieve records by Skyflow ID (single- or multi-table). */ +async function performSecureDataRetrieval(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // Single-table: pass the table and the Skyflow IDs. + const getRequest = new GetRequest('', ['', '']); + + // Multi-table batch: leave table/ids empty and pass per-table records instead. + // Each record can carry its own columns / columnRedactions / uniqueValues. + // const getRequest = new GetRequest('', [], [ + // { tableName: '', ids: [''], columns: [''] }, + // { tableName: '', ids: [], uniqueValues: [{ email: '' }] }, + // ]); + + // Optional: limit columns, redact per column, or look up by unique values. + const getOptions = new GetOptions(); + // getOptions.setFields(['name', 'email']); + // getOptions.setColumnRedactions([{ columnName: 'ssn', redaction: 'MASKED' }]); + // getOptions.setUniqueValues([{ email: 'john@example.com' }]); + + const response = await skyflowClient + .vault('') + .get(getRequest, getOptions); + + // response.records: one GetResponseRecord per result. Each has skyflowId, + // tableName, data, tokens, hashedData, httpCode, error, requestId. + console.log('Records:', response.records); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performSecureDataRetrieval(); diff --git a/flowvault/samples/vault-api/get-tokens.ts b/flowvault/samples/vault-api/get-tokens.ts new file mode 100644 index 00000000..96389d0e --- /dev/null +++ b/flowvault/samples/vault-api/get-tokens.ts @@ -0,0 +1,55 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + GetTokensRequest, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Look up the existing token for a value (deterministic token groups only). */ +async function performGetTokens(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // One record per input value. Only deterministic token groups are supported. + const getTokensRequest = new GetTokensRequest([ + { value: 'john@example.com', tokenGroupName: '' }, + { value: 'jane@example.com', tokenGroupName: '' }, + ]); + + const response = await skyflowClient + .vault('') + .getTokens(getTokensRequest); + + // response.records: one GetTokensResponseRecord per input, in order. Each has + // value, tokenGroupName, token, httpCode, error, requestId. + console.log('Records:', response.records); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performGetTokens(); diff --git a/flowvault/samples/vault-api/insert-records.ts b/flowvault/samples/vault-api/insert-records.ts new file mode 100644 index 00000000..b3e2c4b2 --- /dev/null +++ b/flowvault/samples/vault-api/insert-records.ts @@ -0,0 +1,74 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + InsertRequest, + InsertOptions, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Insert records into one or more tables. */ +async function performSecureDataInsertion(): Promise { + try { + const credentials: Credentials = { + path: '', // or { apiKey } / { token } / { credentialsString } + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', // from the vault URL: https://{clusterId}.vault.skyflowapis.com + env: Env.PROD, // DEV, STAGE, SANDBOX, or PROD + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // One record per entry. Set `table` on a record to insert into a different + // table in the same call (multi-table insert); otherwise the request-level + // table applies. `tokens` enables BYOT (bring your own token). + const insertRequest = new InsertRequest('', [ + { data: { name: 'John Doe', email: 'john@example.com' } }, + // Per-record table + upsert (insert across tables in one call). When any + // record sets its own upsert, the request-level upsert below is ignored. + // { + // data: { name: 'Jane Doe', email: 'jane@example.com' }, + // table: '', + // upsert: { updateType: 'UPDATE', uniqueColumns: ['email'] }, + // }, + ]); + + // Request-level upsert: update when a matching value exists in a unique column. + const insertOptions = new InsertOptions(); + // insertOptions.setUpsert({ updateType: 'UPDATE', uniqueColumns: ['email'] }); + + const response = await skyflowClient + .vault('') + .insert(insertRequest, insertOptions); + + // response.records: one InsertResponseRecord per input, in order. Each has + // tableName, skyflowId, tokens, data, hashedData, httpCode, error, requestId. + for (const record of response.records) { + if (record.error == null) { + console.log(`${record.skyflowId} -> tokens=`, record.tokens); + } else { + console.log(`failed (${record.httpCode}): ${record.error} [requestId=${record.requestId}]`); + } + } + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performSecureDataInsertion(); diff --git a/flowvault/samples/vault-api/query-records.ts b/flowvault/samples/vault-api/query-records.ts new file mode 100644 index 00000000..1f77ba27 --- /dev/null +++ b/flowvault/samples/vault-api/query-records.ts @@ -0,0 +1,56 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + QueryRequest, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Run a read-only SQL SELECT query against the vault. */ +async function performQuery(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // Only SELECT is supported; a call returns at most 25 rows — page with OFFSET. + const queryRequest = new QueryRequest( + "SELECT name, email FROM LIMIT 25", + ); + + const response = await skyflowClient + .vault('') + .query(queryRequest); + + // response.fields: the result rows (one object per row). response.metadata.columns + // lists the returned columns; response.requestId is always set. + console.log('Request ID:', response.requestId); + console.log('Columns:', response.metadata?.columns); + console.log('Rows:', response.fields); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performQuery(); diff --git a/flowvault/samples/vault-api/timeout-and-retry-config-example.ts b/flowvault/samples/vault-api/timeout-and-retry-config-example.ts new file mode 100644 index 00000000..1fd6a8e0 --- /dev/null +++ b/flowvault/samples/vault-api/timeout-and-retry-config-example.ts @@ -0,0 +1,62 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + Credentials, +} from 'skyflow-flowvault-node'; + +/** + * Per-vault timeout and retry configuration. + * + * The flowvault SDK honors these VaultConfig fields (applied to the generated + * client's request layer at client construction): + * timeout overall per-call timeout in seconds, incl. retries + backoff. Default 60. + * maxRetries retry attempts after the first failure (408/429/5xx). Default off. + * vaultUrl override the vault URL derived from clusterId/env. + * + * NOTE: the finer-grained connect/read/write timeouts and retry-delay knobs from + * the Python SDK are fixed inside the Node generated client and are not + * configurable here. + */ +async function performWithTimeoutAndRetry(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + timeout: 30, // seconds + maxRetries: 2, + // vaultUrl: 'https://.skyvault.skyflowapis.com', // optional URL override + }, + ], + logLevel: LogLevel.ERROR, + }); + + console.log('Skyflow client configured with per-vault timeout & retry settings:', skyflowClient); + + // Use the client as usual; requests now fail fast at the configured timeout + // and retry transient 408/429/5xx responses with exponential backoff + jitter. + // const res = await skyflowClient.vault('').query(new QueryRequest('SELECT 1')); + // console.log(res.fields); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performWithTimeoutAndRetry(); diff --git a/flowvault/samples/vault-api/update-record.ts b/flowvault/samples/vault-api/update-record.ts new file mode 100644 index 00000000..23b94c83 --- /dev/null +++ b/flowvault/samples/vault-api/update-record.ts @@ -0,0 +1,62 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + UpdateRequest, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Update existing records by Skyflow ID. */ +async function performSecureDataUpdate(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // Each record needs a skyflowId. Set `table` on a record to target a + // different table in the same call; `tokens` enables BYOT. The optional 3rd + // arg is a request-level updateType: 'UPDATE' (merge) or 'REPLACE' (overwrite). + const updateRequest = new UpdateRequest('', [ + { skyflowId: '', data: { name: 'Jane Doe' } }, + ], 'UPDATE'); + + const response = await skyflowClient + .vault('') + .update(updateRequest); + + // response.records: one UpdateResponseRecord per input, in order. Each has + // skyflowId, tableName, tokens, data, hashedData, httpCode, error, requestId. + for (const record of response.records) { + if (record.error == null) { + console.log(`${record.skyflowId} updated -> tokens=`, record.tokens); + } else { + console.log(`${record.skyflowId} failed (${record.httpCode}): ${record.error}`); + } + } + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performSecureDataUpdate(); diff --git a/flowvault/samples/vault-api/upload-files.ts b/flowvault/samples/vault-api/upload-files.ts new file mode 100644 index 00000000..d06c6667 --- /dev/null +++ b/flowvault/samples/vault-api/upload-files.ts @@ -0,0 +1,64 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + UploadFilesRequest, + Credentials, +} from 'skyflow-flowvault-node'; + +/** Upload files into file columns. */ +async function performUploadFiles(): Promise { + try { + const credentials: Credentials = { + path: '', + }; + + const skyflowClient = new Skyflow({ + vaultConfigs: [ + { + vaultId: '', + clusterId: '', + env: Env.PROD, + credentials, + }, + ], + logLevel: LogLevel.ERROR, + }); + + // Each column takes exactly ONE source: filePath / base64 (needs fileName) / + // file (a Node File object, v20+). Omit skyflowId to CREATE a new record; set + // it to upload into an EXISTING record. + const uploadFilesRequest = new UploadFilesRequest([ + { + tableName: '', + // skyflowId: '', + columns: [ + { column: '', filePath: '' }, + // { column: '', base64: '', fileName: 'doc.pdf' }, + ], + }, + ]); + + const response = await skyflowClient + .vault('') + .uploadFiles(uploadFilesRequest); + + // response.records: one UploadFilesResponseRecord per input record, in order. + // Each has skyflowId, tableName, columns, httpCode, error, requestId. Each + // column result has column, fileName, uploadStatus ("UPLOADED"/"FAILED"/"SKIPPED"), error. + console.log('Records:', JSON.stringify(response.records, null, 2)); + } catch (error) { + if (error instanceof SkyflowError) { + console.error('Skyflow Error:', { + code: error.error?.httpCode, + message: error.message, + details: error.error?.details, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +performUploadFiles(); diff --git a/flowvault/src/client/index.ts b/flowvault/src/client/index.ts new file mode 100644 index 00000000..62fa2c8b --- /dev/null +++ b/flowvault/src/client/index.ts @@ -0,0 +1,224 @@ +import { + AuthInfo, + AuthType, + Credentials, + LogLevel, + MessageType, + SkyflowError, + errorMessages, + getBearerToken, + isExpired, + printLog, + logs, + HTTP_HEADER, + HTTP_STATUS_CODE, +} from 'skyflow-common'; +import { Records } from '../../_generated_/rest/src/api/resources/records/client/Client'; +import { Tokens } from '../../_generated_/rest/src/api/resources/tokens/client/Client'; +import { Files } from '../../_generated_/rest/src/api/resources/files/client/Client'; +import { Query } from '../../_generated_/rest/src/api/resources/query/client/Client'; + +/** + * VaultClient owns the connection details for a single vault: its URL, + * credentials, log level and bearer token. It lazily builds the generated + * resource clients (Records / Tokens / Files / Query), each wired to a + * baseUrl + token supplier, and exposes them as typed accessors. The + * generated top-level `SkyflowClient` wrapper is intentionally not used (see + * notes) — the resource clients take the exact same options and are composed + * directly, mirroring the sibling skyvault SDK. + * + * It also normalizes the v2 error envelope into a common `SkyflowError`. + */ +class VaultClient { + vaultId: string; + + url: string; + + private currentToken = ''; + + private recordsApi?: Records; + + private tokensApi?: Tokens; + + private filesApi?: Files; + + private queryApi?: Query; + + individualCredentials?: Credentials; + + skyflowCredentials?: Credentials; + + logLevel: LogLevel; + + /** Overall per-call timeout in seconds; undefined -> generated default (60s). */ + timeout?: number; + + /** Retry attempts after the first failure; undefined -> generated default. */ + maxRetries?: number; + + authInfo?: AuthInfo; + + private updateTriggered = false; + + constructor( + url: string, + vaultId: string, + individualCredentials?: Credentials, + skyflowCredentials?: Credentials, + logLevel?: LogLevel, + httpConfig?: { timeout?: number; maxRetries?: number }, + ) { + this.url = url; + this.vaultId = vaultId; + this.individualCredentials = individualCredentials; + this.skyflowCredentials = skyflowCredentials; + this.logLevel = logLevel || LogLevel.ERROR; + this.timeout = httpConfig?.timeout; + this.maxRetries = httpConfig?.maxRetries; + } + + updateClientConfig( + url: string, + vaultId: string, + individualCredentials?: Credentials, + skyflowCredentials?: Credentials, + logLevel?: LogLevel, + ): void { + this.updateTriggered = true; + this.url = url; + this.vaultId = vaultId; + this.individualCredentials = individualCredentials; + this.skyflowCredentials = skyflowCredentials; + this.logLevel = logLevel || LogLevel.ERROR; + // Force the resource clients to be rebuilt against the new URL. + this.recordsApi = undefined; + this.tokensApi = undefined; + this.filesApi = undefined; + this.queryApi = undefined; + } + + getLogLevel(): LogLevel { + return this.logLevel; + } + + setLogLevel(logLevel: LogLevel): void { + this.logLevel = logLevel; + } + + updateSkyflowCredentials(credentials?: Credentials): void { + this.updateTriggered = true; + this.skyflowCredentials = credentials; + } + + getCredentials(): Credentials | undefined { + if (this.authInfo?.key && !this.updateTriggered) { + switch (this.authInfo.type) { + case AuthType.API_KEY: + printLog(logs.infoLogs.REUSE_KEY, MessageType.LOG, this.logLevel); + return { apiKey: this.authInfo.key } as Credentials; + case AuthType.TOKEN: + if (!isExpired(this.authInfo.key)) { + printLog(logs.infoLogs.REUSE_BEARER_TOKEN, MessageType.LOG, this.logLevel); + return { token: this.authInfo.key } as Credentials; + } + break; + default: + break; + } + } + this.updateTriggered = false; + return this.individualCredentials ?? this.skyflowCredentials; + } + + /** Resolve a bearer token and make sure the generated clients are ready. */ + async getToken(): Promise { + const authInfo = await getBearerToken(this.getCredentials(), this.logLevel); + this.authInfo = authInfo; + this.currentToken = authInfo.key; + } + + private supplierOptions(): Records.Options { + return { + baseUrl: () => this.url, + token: () => this.currentToken, + }; + } + + get records(): Records { + if (!this.recordsApi) { + this.recordsApi = new Records(this.supplierOptions()); + } + return this.recordsApi; + } + + get tokens(): Tokens { + if (!this.tokensApi) { + this.tokensApi = new Tokens(this.supplierOptions()); + } + return this.tokensApi; + } + + get files(): Files { + if (!this.filesApi) { + this.filesApi = new Files(this.supplierOptions()); + } + return this.filesApi; + } + + get query(): Query { + if (!this.queryApi) { + this.queryApi = new Query(this.supplierOptions()); + } + return this.queryApi; + } + + /** + * Normalize a thrown error (generated SkyflowError envelope, timeout, or a + * plain Error) into a common `SkyflowError`. The v2 envelope is + * `{ error: { requestId?, grpcCode?/grpc_code?, httpCode?/http_code?, + * httpStatus?/http_status?, message, details? } }`. + */ + failureResponse(err: any): SkyflowError { + if (err instanceof SkyflowError) { + return err; + } + + const rawResponse = err?.rawResponse; + const headers = rawResponse?.headers; + const headerRequestId = + headers && headers.get + ? headers.get(HTTP_HEADER.X_REQUEST_ID) || '' + : ''; + + const body = err?.body; + const errorData = (body && typeof body === 'object' ? body.error : undefined) ?? {}; + + const message: string = + errorData.message ?? err?.message ?? errorMessages.ERROR_OCCURRED; + const httpCode = + err?.statusCode ?? + errorData.httpCode ?? + errorData.http_code ?? + HTTP_STATUS_CODE.BAD_REQUEST; + const grpcCode = errorData.grpcCode ?? errorData.grpc_code; + const httpStatus = errorData.httpStatus ?? errorData.http_status; + const details = errorData.details ?? []; + const requestId = errorData.requestId ?? headerRequestId; + + printLog(message, MessageType.ERROR, this.logLevel); + + return new SkyflowError( + { + httpCode, + message, + requestId, + grpcCode, + httpStatus, + details, + }, + [], + ); + } +} + +export default VaultClient; diff --git a/flowvault/src/controller/vault/index.ts b/flowvault/src/controller/vault/index.ts new file mode 100644 index 00000000..f1f2337e --- /dev/null +++ b/flowvault/src/controller/vault/index.ts @@ -0,0 +1,863 @@ +import * as fs from 'fs/promises'; +import * as path from 'path'; +import * as Gen from '../../../_generated_/rest/src/api'; +import { + SDK, + MessageType, + LogLevel, + printLog, + generateSDKMetrics, + removeSDKVersion, + SkyflowError, + logs, +} from 'skyflow-common'; +import VaultClient from '../../client'; +import { FlowVaultOptions } from '../../model/options/base'; +import { RequestContext } from '../../model/request-context'; +import InsertRequest from '../../model/request/insert'; +import InsertOptions from '../../model/options/insert'; +import InsertResponse from '../../model/response/insert'; +import GetRequest from '../../model/request/get'; +import GetOptions from '../../model/options/get'; +import GetResponse from '../../model/response/get'; +import UpdateRequest from '../../model/request/update'; +import UpdateOptions from '../../model/options/update'; +import UpdateResponse from '../../model/response/update'; +import DeleteRequest from '../../model/request/delete'; +import DeleteResponse from '../../model/response/delete'; +import QueryRequest from '../../model/request/query'; +import QueryResponse from '../../model/response/query'; +import DetokenizeRequest from '../../model/request/detokenize'; +import DetokenizeOptions from '../../model/options/detokenize'; +import DeleteOptions from '../../model/options/delete'; +import QueryOptions from '../../model/options/query'; +import GetTokensOptions from '../../model/options/get-tokens'; +import UploadFilesOptions from '../../model/options/upload-files'; +import DeleteFilesOptions from '../../model/options/delete-files'; +import DetokenizeResponse from '../../model/response/detokenize'; +import GetTokensRequest from '../../model/request/get-tokens'; +import GetTokensResponse from '../../model/response/get-tokens'; +import UploadFilesRequest from '../../model/request/upload-files'; +import UploadFilesResponse from '../../model/response/upload-files'; +import DeleteFilesRequest from '../../model/request/delete-files'; +import DeleteFilesResponse from '../../model/response/delete-files'; +import { + InsertResponseRecord, + GetResponseRecord, + UpdateResponseRecord, + DeleteResponseRecord, + DetokenizeResponseRecord, + GetTokensResponseRecord, + UploadFilesColumn, + UploadFilesResponseColumn, + UploadFilesResponseRecord, + DeleteFilesResponseRecord, + UpsertOptions, +} from '../../types'; +import { + validateInsertRequest, + validateGetRequest, + validateUpdateRequest, + validateDeleteRequest, + validateQueryRequest, + validateDetokenizeRequest, + validateGetTokensRequest, + validateUploadFilesRequest, + validateDeleteFilesRequest, +} from '../../validations'; + +interface PreparedColumn { + column: string; + fileName?: string; + bytes: Uint8Array; + contentType: string; +} + +/** + * Operation names handed to a request interceptor via {@link RequestContext}. + * Kept identical to the Python/Java FlowVault SDKs so interceptor logic ports + * across languages. + */ +const Operation = { + INSERT: 'INSERT', + GET: 'GET', + UPDATE: 'UPDATE', + DELETE: 'DELETE', + DETOKENIZE: 'DETOKENIZE', + QUERY: 'QUERY', + GET_TOKENS: 'GET_TOKENS', + UPLOAD_FILES: 'UPLOAD_FILES', + DELETE_FILES: 'DELETE_FILES', +} as const; + +/** + * VaultController holds the 9 unary FlowVault operations. Each validates its + * input, ensures a bearer token, calls the generated client, and maps the + * generated response onto the hand-written Response shape. Whole-call + * failures are normalized to `SkyflowError` and thrown. + */ +class VaultController { + private client: VaultClient; + + constructor(client: VaultClient) { + this.client = client; + } + + private get logLevel(): LogLevel { + return this.client.getLogLevel(); + } + + private sdkHeaders(): Record { + return { [SDK.METRICS_HEADER_KEY]: JSON.stringify(generateSDKMetrics(this.logLevel)) }; + } + + /** + * Request options for every generated call: the SDK metrics header merged with + * any per-call custom headers (from the options' interceptor), plus the vault's + * configured timeout (seconds) and retry count, when set. + */ + private requestOptions(operation: string, callOptions?: FlowVaultOptions): { + headers: Record; + timeoutInSeconds?: number; + maxRetries?: number; + } { + const result: { headers: Record; timeoutInSeconds?: number; maxRetries?: number } = { + headers: { ...this.sdkHeaders(), ...this.customHeaders(operation, callOptions) }, + }; + if (this.client.timeout !== undefined) result.timeoutInSeconds = this.client.timeout; + if (this.client.maxRetries !== undefined) result.maxRetries = this.client.maxRetries; + return result; + } + + /** + * Run the caller's interceptor (if any) against a fresh {@link RequestContext} + * and turn the headers it set into wire headers. Each header key is a + * {@link CustomHeaderKey} whose value is already the header name. + */ + // eslint-disable-next-line class-methods-use-this + private customHeaders(operation: string, callOptions?: FlowVaultOptions): Record { + const interceptor = callOptions?.getInterceptor(); + if (!interceptor) return {}; + const context = new RequestContext(operation); + interceptor(context); + const headers: Record = {}; + context.headers.forEach((value, key) => { + headers[key] = value; + }); + return headers; + } + + // eslint-disable-next-line class-methods-use-this + private getRequestId(rawResponse: any): string | undefined { + const headers = rawResponse?.headers; + if (headers && headers.get) { + return headers.get('x-request-id') || undefined; + } + return undefined; + } + + private handleError(error: unknown): SkyflowError { + if (error instanceof Error) { + printLog(removeSDKVersion(error.message), MessageType.ERROR, this.logLevel); + } + return this.client.failureResponse(error); + } + + private static toGenUpsert(upsert: UpsertOptions): Gen.Upsert { + return { + updateType: upsert.updateType as Gen.UpsertUpdateType, + uniqueColumns: upsert.uniqueColumns, + }; + } + + // Per-record failures can come back with a 4xx HTTP status whose body carries a + // records/response array (e.g. a delete/get/update of a non-existent id). Pull + // those out so they surface as a normal response instead of a thrown error. + private static unaryErrorRecords(error: unknown): unknown[] | null { + const body = (error as { body?: { records?: unknown; response?: unknown } })?.body; + if (body && typeof body === 'object') { + if (Array.isArray(body.records)) return body.records; + if (Array.isArray(body.response)) return body.response; + } + return null; + } + + // Surface per-record failures from an error body as a normal response (matches + // Java/Python flowvault); re-throw genuine whole-call errors. + private unaryError(error: unknown, build: (records: unknown[], requestId?: string) => T): T { + const errorRecords = VaultController.unaryErrorRecords(error); + if (errorRecords) { + const requestId = this.getRequestId((error as { rawResponse?: unknown }).rawResponse); + return build(errorRecords, requestId); + } + throw this.handleError(error); + } + + // ---------------------------------------------------------------- insert + async insert(request: InsertRequest, options?: InsertOptions): Promise { + try { + printLog(logs.infoLogs.INSERT_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_INSERT_INPUT, MessageType.LOG, this.logLevel); + validateInsertRequest(request, options, this.logLevel); + await this.client.getToken(); + + // When any record carries its own table, send a per-record tableName so a + // single insert can target multiple tables (matches Java/Python flowvault). + const insertPerRecordTable = request.records.some( + (record) => record.table !== undefined && record.table !== null, + ); + const requestUpsert = options?.getUpsert(); + // When any record carries its own upsert, send per-record upsert and drop + // the request-level one (matches Java/Python flowvault). + const insertPerRecordUpsert = request.records.some( + (record) => record.upsert !== undefined && record.upsert !== null, + ); + const genReq: Gen.InsertRequest = { + vaultID: this.client.vaultId, + records: request.records.map((record) => { + const genRecord: Gen.InsertRecordData = { data: record.data }; + if (record.tokens) genRecord.tokens = record.tokens; // BYOT + if (insertPerRecordTable) genRecord.tableName = record.table ?? request.tableName; + if (insertPerRecordUpsert) { + const recordUpsert = record.upsert ?? requestUpsert; + if (recordUpsert) genRecord.upsert = VaultController.toGenUpsert(recordUpsert); + } + return genRecord; + }), + } as Gen.InsertRequest; + + // The API requires tableName in exactly ONE place. Send it at the request + // level only when no record carries its own; otherwise it is per-record. + if (!insertPerRecordTable) { + genReq.tableName = request.tableName; + } + + if (!insertPerRecordUpsert && requestUpsert) { + genReq.upsert = VaultController.toGenUpsert(requestUpsert); + } + + const { data, rawResponse } = await this.client.records + .insertRecords(genReq, this.requestOptions(Operation.INSERT, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.INSERT_DATA_SUCCESS, MessageType.LOG, this.logLevel); + + const records = (data.records ?? []).map((record) => + this.mapInsertRecord(record, requestId), + ); + return new InsertResponse({ records }); + } catch (error) { + return this.unaryError(error, (errorRecords, requestId) => + new InsertResponse({ + records: errorRecords.map((record) => this.mapInsertRecord(record as Gen.RecordResponseObject, requestId)), + })); + } + } + + // eslint-disable-next-line class-methods-use-this + private mapInsertRecord( + record: Gen.RecordResponseObject, + requestId?: string, + ): InsertResponseRecord { + const hasError = record.error !== undefined && record.error !== null; + const result: InsertResponseRecord = { + tableName: record.tableName ?? null, + skyflowId: record.skyflowID ?? null, + tokens: record.tokens ?? null, + data: record.data ?? null, + hashedData: record.hashedData ?? null, + httpCode: record.httpCode, + error: hasError ? record.error : null, + }; + if (hasError) { + result.requestId = requestId ?? null; + } + return result; + } + + // ------------------------------------------------------------------- get + async get(request: GetRequest, options?: GetOptions): Promise { + try { + printLog(logs.infoLogs.GET_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_GET_INPUT, MessageType.LOG, this.logLevel); + validateGetRequest(request, options, this.logLevel); + await this.client.getToken(); + + const genReq: Gen.GetRequest = { vaultID: this.client.vaultId }; + if (request.records && request.records.length > 0) { + genReq.records = request.records.map((record) => { + const genRecord: Gen.GetRequestData = { + tableName: record.tableName, + skyflowIDs: record.ids, + }; + if (record.columns) genRecord.columns = record.columns; + if (record.columnRedactions) { + genRecord.columnRedactions = record.columnRedactions.map((redaction) => ({ + columnName: redaction.columnName, + redaction: redaction.redaction, + })); + } + if (record.uniqueValues) { + genRecord.uniqueValues = record.uniqueValues.map((data) => ({ data })); + } + return genRecord; + }); + } else { + genReq.tableName = request.table; + genReq.skyflowIDs = request.ids; + } + + const fields = options?.getFields(); + if (fields) genReq.columns = fields; + const redactions = options?.getColumnRedactions(); + if (redactions) { + genReq.columnRedactions = redactions.map((redaction) => ({ + columnName: redaction.columnName, + redaction: redaction.redaction, + })); + } + const uniqueValues = options?.getUniqueValues(); + if (uniqueValues) { + genReq.uniqueValues = uniqueValues.map((data) => ({ data })); + } + const limit = options?.getLimit(); + if (limit !== undefined) genReq.limit = limit; + const offset = options?.getOffset(); + if (offset !== undefined) genReq.offset = offset; + + const { data, rawResponse } = await this.client.records + .getRecords(genReq, this.requestOptions(Operation.GET, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.GET_SUCCESS, MessageType.LOG, this.logLevel); + + const records = (data.records ?? []).map((record) => this.mapGetRecord(record, requestId)); + return new GetResponse({ records }); + } catch (error) { + return this.unaryError(error, (errorRecords, requestId) => + new GetResponse({ + records: errorRecords.map((record) => this.mapGetRecord(record as Gen.RecordResponseObject, requestId)), + })); + } + } + + // eslint-disable-next-line class-methods-use-this + private mapGetRecord(record: Gen.RecordResponseObject, requestId?: string): GetResponseRecord { + const hasError = record.error !== undefined && record.error !== null; + const result: GetResponseRecord = { + skyflowId: record.skyflowID ?? null, + tableName: record.tableName ?? null, + data: record.data ?? null, + hashedData: record.hashedData ?? null, + tokens: record.tokens ?? null, + httpCode: record.httpCode, + error: hasError ? record.error : null, + }; + if (hasError) { + result.requestId = requestId ?? null; + } + return result; + } + + // ---------------------------------------------------------------- update + async update(request: UpdateRequest, options?: UpdateOptions): Promise { + try { + printLog(logs.infoLogs.UPDATE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_UPDATE_INPUT, MessageType.LOG, this.logLevel); + validateUpdateRequest(request, options, this.logLevel); + await this.client.getToken(); + + const updatePerRecordTable = request.records.some( + (record) => record.table !== undefined && record.table !== null, + ); + const genReq: Gen.UpdateRequest = { + vaultID: this.client.vaultId, + records: request.records.map((record) => { + const genRecord: Gen.UpdateRecordData = { skyflowID: record.skyflowId, data: record.data }; + if (record.tokens) genRecord.tokens = record.tokens; // BYOT + if (updatePerRecordTable) genRecord.tableName = record.table ?? request.tableName; + return genRecord; + }), + } as Gen.UpdateRequest; + // Table in exactly ONE place — request level only when not per-record. + if (!updatePerRecordTable) { + genReq.tableName = request.tableName; + } + if (request.updateType) { + genReq.updateType = request.updateType as Gen.UpdateRequestUpdateType; + } + + const { data, rawResponse } = await this.client.records + .updateRecords(genReq, this.requestOptions(Operation.UPDATE, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.UPDATE_SUCCESS, MessageType.LOG, this.logLevel); + + const records = (data.records ?? []).map((record) => + this.mapUpdateRecord(record, requestId), + ); + return new UpdateResponse({ records }); + } catch (error) { + return this.unaryError(error, (errorRecords, requestId) => + new UpdateResponse({ + records: errorRecords.map((record) => this.mapUpdateRecord(record as Gen.RecordResponseObject, requestId)), + })); + } + } + + // eslint-disable-next-line class-methods-use-this + private mapUpdateRecord( + record: Gen.RecordResponseObject, + requestId?: string, + ): UpdateResponseRecord { + const hasError = record.error !== undefined && record.error !== null; + const result: UpdateResponseRecord = { + skyflowId: record.skyflowID ?? null, + tableName: record.tableName ?? null, + tokens: record.tokens ?? null, + data: record.data ?? null, + hashedData: record.hashedData ?? null, + httpCode: record.httpCode, + error: hasError ? record.error : null, + }; + if (hasError) { + result.requestId = requestId ?? null; + } + return result; + } + + // ---------------------------------------------------------------- delete + async delete(request: DeleteRequest, options?: DeleteOptions): Promise { + try { + printLog(logs.infoLogs.DELETE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_DELETE_INPUT, MessageType.LOG, this.logLevel); + validateDeleteRequest(request, this.logLevel); + await this.client.getToken(); + + const genReq: Gen.DeleteRequest = { + vaultID: this.client.vaultId, + tableName: request.table, + }; + if (request.ids && request.ids.length > 0) { + genReq.skyflowIDs = request.ids; + } + if (request.uniqueValues && request.uniqueValues.length > 0) { + // Each { col: value, ... } map becomes one wire { data: {...} } entry. + genReq.uniqueValues = request.uniqueValues.map((data) => ({ data })); + } + + const { data, rawResponse } = await this.client.records + .deleteRecords(genReq, this.requestOptions(Operation.DELETE, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.DELETE_SUCCESS, MessageType.LOG, this.logLevel); + + const records = (data.records ?? []).map((record) => + this.mapDeleteRecord(record, requestId), + ); + return new DeleteResponse({ records }); + } catch (error) { + return this.unaryError(error, (errorRecords, requestId) => + new DeleteResponse({ + records: errorRecords.map((record) => this.mapDeleteRecord(record as Gen.DeleteResponseObject, requestId)), + })); + } + } + + // eslint-disable-next-line class-methods-use-this + private mapDeleteRecord( + record: Gen.DeleteResponseObject, + requestId?: string, + ): DeleteResponseRecord { + const hasError = record.error !== undefined && record.error !== null; + const result: DeleteResponseRecord = { + skyflowId: record.skyflowID ?? null, + httpCode: record.httpCode, + error: hasError ? record.error : null, + }; + if (hasError) { + result.requestId = requestId ?? null; + } + return result; + } + + // ----------------------------------------------------------------- query + async query(request: QueryRequest, options?: QueryOptions): Promise { + try { + printLog(logs.infoLogs.QUERY_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_QUERY_INPUT, MessageType.LOG, this.logLevel); + validateQueryRequest(request, this.logLevel); + await this.client.getToken(); + + const genReq: Gen.ExecuteQueryRequest = { + vaultID: this.client.vaultId, + query: request.query, + }; + + const { data, rawResponse } = await this.client.query + .executeQuery(genReq, this.requestOptions(Operation.QUERY, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse) ?? ''; + printLog(logs.infoLogs.QUERY_SUCCESS, MessageType.LOG, this.logLevel); + + const fields = (data.records ?? []).map((record) => record.data ?? {}); + // NOTE: generated ExecuteQueryResponse has no per-record errors block. + return new QueryResponse({ + fields, + errors: null, + metadata: { columns: data.metadata?.columns ?? [] }, + requestId, + }); + } catch (error) { + throw this.handleError(error); + } + } + + // ------------------------------------------------------------ detokenize + async detokenize( + request: DetokenizeRequest, + options?: DetokenizeOptions, + ): Promise { + try { + printLog(logs.infoLogs.DETOKENIZE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_DETOKENIZE_INPUT, MessageType.LOG, this.logLevel); + validateDetokenizeRequest(request, options, this.logLevel); + await this.client.getToken(); + + const genReq: Gen.DetokenizeRequest = { + vaultID: this.client.vaultId, + tokens: request.tokens, + }; + const redactions = request.tokenGroupRedactions; + if (redactions) { + genReq.tokenGroupRedactions = redactions.map((redaction) => ({ + tokenGroupName: redaction.tokenGroupName, + redaction: redaction.redaction, + })); + } + + const { data, rawResponse } = await this.client.tokens + .detokenize(genReq, this.requestOptions(Operation.DETOKENIZE, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.DETOKENIZE_SUCCESS, MessageType.LOG, this.logLevel); + + // NOTE: generated DetokenizeResponse exposes records under `response`. + const records = (data.response ?? []).map((record) => + this.mapDetokenizeRecord(record, requestId), + ); + return new DetokenizeResponse({ records }); + } catch (error) { + return this.unaryError(error, (errorRecords, requestId) => + new DetokenizeResponse({ + records: errorRecords.map((record) => this.mapDetokenizeRecord(record as Gen.DetokenizeResponseObject, requestId)), + })); + } + } + + // eslint-disable-next-line class-methods-use-this + private mapDetokenizeRecord( + record: Gen.DetokenizeResponseObject, + requestId?: string, + ): DetokenizeResponseRecord { + const hasError = record.error !== undefined && record.error !== null; + const metadata = record.metadata as Record | undefined; + const result: DetokenizeResponseRecord = { + token: record.token, + tokenGroupName: record.tokenGroupName ?? null, + value: record.value, + metadata: metadata + ? { + skyflowId: (metadata.skyflowID ?? metadata.skyflowId ?? null) as string | null, + tableName: (metadata.tableName ?? null) as string | null, + } + : null, + httpCode: record.httpCode ?? 200, + error: hasError ? record.error : null, + }; + if (hasError) { + result.requestId = requestId ?? null; + } + return result; + } + + // ------------------------------------------------------------- getTokens + async getTokens(request: GetTokensRequest, options?: GetTokensOptions): Promise { + try { + printLog(logs.infoLogs.TOKENIZE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_TOKENIZE_INPUT, MessageType.LOG, this.logLevel); + validateGetTokensRequest(request, this.logLevel); + await this.client.getToken(); + + const genReq: Gen.GetTokensFromValuesRequest = { + vaultID: this.client.vaultId, + records: request.records.map((record) => ({ + value: record.value, + tokenGroupName: record.tokenGroupName, + })), + }; + + const { data, rawResponse } = await this.client.tokens + .getTokens(genReq, this.requestOptions(Operation.GET_TOKENS, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.TOKENIZE_SUCCESS, MessageType.LOG, this.logLevel); + + const records = (data.records ?? []).map((record) => + this.mapGetTokensRecord(record, requestId), + ); + return new GetTokensResponse({ records }); + } catch (error) { + return this.unaryError(error, (errorRecords, requestId) => + new GetTokensResponse({ + records: errorRecords.map((record) => this.mapGetTokensRecord(record as Gen.TokenizeResponseObject, requestId)), + })); + } + } + + // eslint-disable-next-line class-methods-use-this + private mapGetTokensRecord( + record: Gen.TokenizeResponseObject, + requestId?: string, + ): GetTokensResponseRecord { + const hasError = record.error !== undefined && record.error !== null; + const result: GetTokensResponseRecord = { + value: record.value, + tokenGroupName: record.tokenGroupName, + token: record.token ?? null, + httpCode: record.httpCode ?? 200, + error: hasError ? record.error : null, + }; + if (hasError) { + result.requestId = requestId ?? null; + } + return result; + } + + // ----------------------------------------------------------- uploadFiles + async uploadFiles(request: UploadFilesRequest, options?: UploadFilesOptions): Promise { + try { + printLog(logs.infoLogs.UPLOAD_FILE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_FILE_UPLOAD_INPUT, MessageType.LOG, this.logLevel); + validateUploadFilesRequest(request, undefined, this.logLevel); + await this.client.getToken(); + + // Phase A: prepare file bytes and request signed upload URLs. + const prepared: PreparedColumn[][] = []; + for (let i = 0; i < request.records.length; i += 1) { + // eslint-disable-next-line no-await-in-loop + prepared.push(await Promise.all(request.records[i].columns.map((c) => this.prepareColumn(c)))); + } + + const genReq: Gen.FileUploadRequest = { + vaultID: this.client.vaultId, + records: request.records.map((record, index) => { + const genRecord: Gen.FileUploadRecord = { + tableName: record.tableName, + columns: prepared[index].map((column) => { + const col: Gen.FileUploadColumn = { column: column.column }; + if (column.fileName) col.fileName = column.fileName; + return col; + }), + }; + if (record.skyflowId) genRecord.skyflowID = record.skyflowId; + return genRecord; + }), + }; + + const { data, rawResponse } = await this.client.files + .uploadFiles(genReq, this.requestOptions(Operation.UPLOAD_FILES, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.FILE_UPLOAD_DATA_SUCCESS, MessageType.LOG, this.logLevel); + + // Phase B: PUT each file's bytes to its signed URL. + const records: UploadFilesResponseRecord[] = []; + const responseRecords = data.records ?? []; + for (let i = 0; i < responseRecords.length; i += 1) { + const responseRecord = responseRecords[i]; + const preparedColumns = prepared[i] ?? []; + // eslint-disable-next-line no-await-in-loop + const columns = await this.uploadColumns(responseRecord, preparedColumns); + const hasError = responseRecord.error !== undefined && responseRecord.error !== null; + const record: UploadFilesResponseRecord = { + skyflowId: responseRecord.skyflowID ?? null, + tableName: responseRecord.tableName ?? null, + columns, + httpCode: responseRecord.httpCode, + error: hasError ? responseRecord.error : null, + }; + if (hasError) { + record.requestId = requestId ?? null; + } + records.push(record); + } + + return new UploadFilesResponse({ records }); + } catch (error) { + throw this.handleError(error); + } + } + + private async uploadColumns( + responseRecord: Gen.FileUploadResponseObject, + preparedColumns: PreparedColumn[], + ): Promise { + const signedUrls = (responseRecord.data ?? {}) as Record; + const recordErrored = + responseRecord.error !== undefined && responseRecord.error !== null; + + const results = await Promise.all( + preparedColumns.map(async (column) => { + const result: UploadFilesResponseColumn = { + column: column.column, + fileName: column.fileName ?? null, + uploadStatus: 'SKIPPED', + }; + if (recordErrored) { + result.error = responseRecord.error; + return result; + } + const signedUrl = signedUrls[column.column]; + if (typeof signedUrl !== 'string' || signedUrl.length === 0) { + // No signed URL returned for this column -> SKIPPED, no error + // (matches the Python flowvault SDK). + return result; + } + try { + const putResponse = await fetch(signedUrl, { + method: 'PUT', + headers: { 'Content-Type': column.contentType }, + body: column.bytes, + }); + if (putResponse.ok) { + result.uploadStatus = 'UPLOADED'; + } else { + result.uploadStatus = 'FAILED'; + result.error = `Upload failed with status ${putResponse.status}.`; + } + } catch (putError) { + result.uploadStatus = 'FAILED'; + result.error = putError instanceof Error ? putError.message : String(putError); + } + return result; + }), + ); + return results; + } + + private async prepareColumn(column: UploadFilesColumn): Promise { + let bytes: Uint8Array = new Uint8Array(); + let fileName = column.fileName; + if (column.filePath) { + const buffer = await fs.readFile(column.filePath); + bytes = new Uint8Array(buffer); + if (!fileName) fileName = path.basename(column.filePath); + } else if (column.base64 !== undefined) { + bytes = new Uint8Array(Buffer.from(column.base64, 'base64')); + } else if (column.file) { + const arrayBuffer = await column.file.arrayBuffer(); + bytes = new Uint8Array(arrayBuffer); + if (!fileName) fileName = column.file.name; + } + return { + column: column.column, + fileName, + bytes, + contentType: this.inferContentType(fileName), + }; + } + + // eslint-disable-next-line class-methods-use-this + private inferContentType(fileName?: string): string { + const ext = (fileName?.split('.').pop() || '').toLowerCase(); + const map: Record = { + pdf: 'application/pdf', + png: 'image/png', + jpg: 'image/jpeg', + jpeg: 'image/jpeg', + gif: 'image/gif', + txt: 'text/plain', + csv: 'text/csv', + json: 'application/json', + doc: 'application/msword', + docx: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', + xls: 'application/vnd.ms-excel', + xlsx: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + }; + return map[ext] || 'application/octet-stream'; + } + + // ----------------------------------------------------------- deleteFiles + async deleteFiles(request: DeleteFilesRequest, options?: DeleteFilesOptions): Promise { + try { + printLog(logs.infoLogs.UPLOAD_FILE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_FILE_UPLOAD_INPUT, MessageType.LOG, this.logLevel); + validateDeleteFilesRequest(request, this.logLevel); + await this.client.getToken(); + + const genReq: Gen.FileDeleteRequest = { + vaultID: this.client.vaultId, + records: request.records.map((record) => { + const genRecord: Gen.FileDeleteRecord = { + tableName: record.tableName, + columns: record.columns, + }; + if (record.skyflowId) genRecord.skyflowID = record.skyflowId; + if (record.uniqueValues) { + genRecord.uniqueValues = record.uniqueValues.map((data) => ({ data })); + } + return genRecord; + }), + }; + + const { data, rawResponse } = await this.client.files + .deleteFiles(genReq, this.requestOptions(Operation.DELETE_FILES, options)) + .withRawResponse(); + const requestId = this.getRequestId(rawResponse); + printLog(logs.infoLogs.FILE_UPLOAD_DATA_SUCCESS, MessageType.LOG, this.logLevel); + + const records = (data.records ?? []).map((record) => + this.mapDeleteFilesRecord(record, requestId), + ); + return new DeleteFilesResponse({ records }); + } catch (error) { + return this.unaryError(error, (errorRecords, requestId) => + new DeleteFilesResponse({ + records: errorRecords.map((record) => this.mapDeleteFilesRecord(record as Gen.FileDeleteResponseObject, requestId)), + })); + } + } + + // eslint-disable-next-line class-methods-use-this + private mapDeleteFilesRecord( + record: Gen.FileDeleteResponseObject, + requestId?: string, + ): DeleteFilesResponseRecord { + const hasError = record.error !== undefined && record.error !== null; + const columnMap = record.data as Record | undefined; + const result: DeleteFilesResponseRecord = { + skyflowId: record.skyflowID ?? null, + tableName: record.tableName ?? null, + // Status is parsed from each column's response value (object -> .status, + // else the value itself), mirroring the Python flowvault SDK. + columns: columnMap + ? Object.entries(columnMap).map(([column, value]) => ({ + column, + status: + value !== null && typeof value === 'object' + ? (((value as Record).status as string) ?? null) + : ((value as string) ?? null), + })) + : null, + httpCode: record.httpCode, + error: hasError ? record.error : null, + }; + if (hasError) { + result.requestId = requestId ?? null; + } + return result; + } +} + +export default VaultController; diff --git a/flowvault/src/enums/custom-header-key.ts b/flowvault/src/enums/custom-header-key.ts new file mode 100644 index 00000000..052216cb --- /dev/null +++ b/flowvault/src/enums/custom-header-key.ts @@ -0,0 +1,14 @@ +/** + * The custom request-header keys the FlowVault SDK supports. + * + * Only these headers may be attached to an outgoing request (through a + * {@link RequestInterceptor}). Each enum value is the header name sent on the + * wire, so `CustomHeaderKey.REQUEST_ID_HEADER` maps to `x-request-id`. + */ +export enum CustomHeaderKey { + SKYFLOW_ACCOUNT_ID = 'x-skyflow-account-id', + SKYFLOW_ACCOUNT_NAME = 'x-skyflow-account-name', + REQUEST_ID_HEADER = 'x-request-id', +} + +export default CustomHeaderKey; diff --git a/flowvault/src/index.ts b/flowvault/src/index.ts new file mode 100644 index 00000000..bfea3a6b --- /dev/null +++ b/flowvault/src/index.ts @@ -0,0 +1,73 @@ +// Re-export the shared public surface from skyflow-common so consumers can +// import everything (credentials, config, errors, enums, service-account +// helpers, validators, base classes, etc.) from skyflow-flowvault-node. +export * from 'skyflow-common'; + +// FlowVault facade. +export { default as Skyflow, default } from './skyflow'; + +// Request classes. +export { default as InsertRequest } from './model/request/insert'; +export { default as GetRequest } from './model/request/get'; +export { default as UpdateRequest } from './model/request/update'; +export { default as DeleteRequest } from './model/request/delete'; +export { default as QueryRequest } from './model/request/query'; +export { default as DetokenizeRequest } from './model/request/detokenize'; +export { default as GetTokensRequest } from './model/request/get-tokens'; +export { default as UploadFilesRequest } from './model/request/upload-files'; +export { default as DeleteFilesRequest } from './model/request/delete-files'; + +// Custom request headers (typed interceptor + supported header keys). +export { CustomHeaderKey } from './enums/custom-header-key'; +export { RequestContext } from './model/request-context'; +export type { RequestInterceptor } from './model/request-context'; +export { FlowVaultOptions } from './model/options/base'; + +// Options classes. +export { default as InsertOptions } from './model/options/insert'; +export { default as GetOptions } from './model/options/get'; +export { default as UpdateOptions } from './model/options/update'; +export { default as DeleteOptions } from './model/options/delete'; +export { default as QueryOptions } from './model/options/query'; +export { default as DetokenizeOptions } from './model/options/detokenize'; +export { default as GetTokensOptions } from './model/options/get-tokens'; +export { default as UploadFilesOptions } from './model/options/upload-files'; +export { default as DeleteFilesOptions } from './model/options/delete-files'; + +// Response classes. +export { default as InsertResponse } from './model/response/insert'; +export { default as GetResponse } from './model/response/get'; +export { default as UpdateResponse } from './model/response/update'; +export { default as DeleteResponse } from './model/response/delete'; +export { default as QueryResponse } from './model/response/query'; +export { default as DetokenizeResponse } from './model/response/detokenize'; +export { default as GetTokensResponse } from './model/response/get-tokens'; +export { default as UploadFilesResponse } from './model/response/upload-files'; +export { default as DeleteFilesResponse } from './model/response/delete-files'; + +// Record / helper types. +export type { GetRecord } from './model/request/get'; +export type { + InsertRecord, + UpsertOptions, + ColumnRedaction, + UpdateRecord, + TokenGroupRedaction, + GetTokensRecord, + UploadFilesColumn, + UploadFilesRecord, + DeleteFilesRecord, + UploadStatus, + InsertResponseRecord, + GetResponseRecord, + UpdateResponseRecord, + DeleteResponseRecord, + DetokenizeResponseRecord, + GetTokensResponseRecord, + UploadFilesResponseColumn, + UploadFilesResponseRecord, + DeleteFilesResponseColumn, + DeleteFilesResponseRecord, + QueryResponseMetadata, + FlowVaultRecordError, +} from './types'; diff --git a/flowvault/src/model/options/base.ts b/flowvault/src/model/options/base.ts new file mode 100644 index 00000000..4cecc173 --- /dev/null +++ b/flowvault/src/model/options/base.ts @@ -0,0 +1,24 @@ +import { RequestInterceptor } from '../request-context'; + +/** + * Base class for every FlowVault `*Options`. + * + * It carries an optional {@link RequestInterceptor} — the one supported way to + * attach custom headers to a call. The interceptor may only set the headers in + * {@link CustomHeaderKey}; arbitrary headers are not supported. + */ +export class FlowVaultOptions { + private interceptor?: RequestInterceptor; + + /** Register an interceptor that attaches supported custom headers per request. */ + setInterceptor(interceptor: RequestInterceptor): void { + this.interceptor = interceptor; + } + + /** The registered request interceptor, if any. */ + getInterceptor(): RequestInterceptor | undefined { + return this.interceptor; + } +} + +export default FlowVaultOptions; diff --git a/flowvault/src/model/options/delete-files/index.ts b/flowvault/src/model/options/delete-files/index.ts new file mode 100644 index 00000000..01b78d18 --- /dev/null +++ b/flowvault/src/model/options/delete-files/index.ts @@ -0,0 +1,10 @@ +import { FlowVaultOptions } from '../base'; + +/** Options for a deleteFiles operation. */ +class DeleteFilesOptions extends FlowVaultOptions { + constructor() { + super(); + } +} + +export default DeleteFilesOptions; diff --git a/flowvault/src/model/options/delete/index.ts b/flowvault/src/model/options/delete/index.ts new file mode 100644 index 00000000..35eb16f6 --- /dev/null +++ b/flowvault/src/model/options/delete/index.ts @@ -0,0 +1,10 @@ +import { FlowVaultOptions } from '../base'; + +/** Options for a delete operation. */ +class DeleteOptions extends FlowVaultOptions { + constructor() { + super(); + } +} + +export default DeleteOptions; diff --git a/flowvault/src/model/options/detokenize/index.ts b/flowvault/src/model/options/detokenize/index.ts new file mode 100644 index 00000000..6821f0f5 --- /dev/null +++ b/flowvault/src/model/options/detokenize/index.ts @@ -0,0 +1,10 @@ +import { FlowVaultOptions } from '../base'; + +/** Options for a detokenize operation. */ +class DetokenizeOptions extends FlowVaultOptions { + constructor() { + super(); + } +} + +export default DetokenizeOptions; diff --git a/flowvault/src/model/options/get-tokens/index.ts b/flowvault/src/model/options/get-tokens/index.ts new file mode 100644 index 00000000..1c2bd063 --- /dev/null +++ b/flowvault/src/model/options/get-tokens/index.ts @@ -0,0 +1,10 @@ +import { FlowVaultOptions } from '../base'; + +/** Options for a getTokens operation. */ +class GetTokensOptions extends FlowVaultOptions { + constructor() { + super(); + } +} + +export default GetTokensOptions; diff --git a/flowvault/src/model/options/get/index.ts b/flowvault/src/model/options/get/index.ts new file mode 100644 index 00000000..9815b619 --- /dev/null +++ b/flowvault/src/model/options/get/index.ts @@ -0,0 +1,57 @@ +import { FlowVaultOptions } from '../base'; +import { ColumnRedaction } from '../../../types'; + +/** Options for a get operation. */ +class GetOptions extends FlowVaultOptions { + private fields?: string[]; + private columnRedactions?: ColumnRedaction[]; + private uniqueValues?: Record[]; + private limit?: number; + private offset?: number; + + constructor() { + super(); + } + + setFields(fields: string[]): void { + this.fields = fields; + } + + getFields(): string[] | undefined { + return this.fields; + } + + setColumnRedactions(columnRedactions: ColumnRedaction[]): void { + this.columnRedactions = columnRedactions; + } + + getColumnRedactions(): ColumnRedaction[] | undefined { + return this.columnRedactions; + } + + setUniqueValues(uniqueValues: Record[]): void { + this.uniqueValues = uniqueValues; + } + + getUniqueValues(): Record[] | undefined { + return this.uniqueValues; + } + + setLimit(limit: number): void { + this.limit = limit; + } + + getLimit(): number | undefined { + return this.limit; + } + + setOffset(offset: number): void { + this.offset = offset; + } + + getOffset(): number | undefined { + return this.offset; + } +} + +export default GetOptions; diff --git a/flowvault/src/model/options/insert/index.ts b/flowvault/src/model/options/insert/index.ts new file mode 100644 index 00000000..af7d6f33 --- /dev/null +++ b/flowvault/src/model/options/insert/index.ts @@ -0,0 +1,21 @@ +import { FlowVaultOptions } from '../base'; +import { UpsertOptions } from '../../../types'; + +/** Options for an insert operation. */ +class InsertOptions extends FlowVaultOptions { + private upsert?: UpsertOptions; + + constructor() { + super(); + } + + setUpsert(upsert: UpsertOptions): void { + this.upsert = upsert; + } + + getUpsert(): UpsertOptions | undefined { + return this.upsert; + } +} + +export default InsertOptions; diff --git a/flowvault/src/model/options/query/index.ts b/flowvault/src/model/options/query/index.ts new file mode 100644 index 00000000..5e1d8404 --- /dev/null +++ b/flowvault/src/model/options/query/index.ts @@ -0,0 +1,10 @@ +import { FlowVaultOptions } from '../base'; + +/** Options for a query operation. */ +class QueryOptions extends FlowVaultOptions { + constructor() { + super(); + } +} + +export default QueryOptions; diff --git a/flowvault/src/model/options/update/index.ts b/flowvault/src/model/options/update/index.ts new file mode 100644 index 00000000..07df3cf8 --- /dev/null +++ b/flowvault/src/model/options/update/index.ts @@ -0,0 +1,10 @@ +import { FlowVaultOptions } from '../base'; + +/** Options for an update operation. */ +class UpdateOptions extends FlowVaultOptions { + constructor() { + super(); + } +} + +export default UpdateOptions; diff --git a/flowvault/src/model/options/upload-files/index.ts b/flowvault/src/model/options/upload-files/index.ts new file mode 100644 index 00000000..706ee51e --- /dev/null +++ b/flowvault/src/model/options/upload-files/index.ts @@ -0,0 +1,10 @@ +import { FlowVaultOptions } from '../base'; + +/** Options for an uploadFiles operation. */ +class UploadFilesOptions extends FlowVaultOptions { + constructor() { + super(); + } +} + +export default UploadFilesOptions; diff --git a/flowvault/src/model/request-context/index.ts b/flowvault/src/model/request-context/index.ts new file mode 100644 index 00000000..bb19eed6 --- /dev/null +++ b/flowvault/src/model/request-context/index.ts @@ -0,0 +1,55 @@ +import { CustomHeaderKey } from '../../enums/custom-header-key'; + +/** Reported when the caller's request was not split into batches. */ +const NOT_BATCHED = -1; + +/** + * The context handed to a {@link RequestInterceptor} for each outgoing request. + * + * Use {@link RequestContext.addHeader} to attach one of the supported custom + * headers (see {@link CustomHeaderKey}). `operation` names the vault operation + * (`INSERT`, `GET`, ...), so an interceptor can vary a header per operation. + */ +export class RequestContext { + readonly operation: string; + + /** + * Zero-based position of this batch within the caller's request, or `-1` when + * the operation was not batched. FlowVault's unary operations are never + * batched, so this is always `-1` today; it mirrors the other SDKs' context. + */ + readonly batchIndex: number; + + /** Total number of batches the request was split into, or `-1` when not batched. */ + readonly totalBatches: number; + + private readonly headerMap: Map = new Map(); + + constructor( + operation: string, + batchIndex: number = NOT_BATCHED, + totalBatches: number = NOT_BATCHED, + ) { + this.operation = operation; + this.batchIndex = batchIndex; + this.totalBatches = totalBatches; + } + + /** Attach a supported custom header to the outgoing request. */ + addHeader(key: CustomHeaderKey, value: string): void { + this.headerMap.set(key, value); + } + + /** A copy of the headers the interceptor added, keyed by {@link CustomHeaderKey}. */ + get headers(): Map { + return new Map(this.headerMap); + } +} + +/** + * A hook invoked once per outgoing request. Attach supported custom headers to + * the request by calling {@link RequestContext.addHeader} on the context. + */ +export type RequestInterceptor = (context: RequestContext) => void; + +export default RequestContext; diff --git a/flowvault/src/model/request/delete-files/index.ts b/flowvault/src/model/request/delete-files/index.ts new file mode 100644 index 00000000..55aad34e --- /dev/null +++ b/flowvault/src/model/request/delete-files/index.ts @@ -0,0 +1,22 @@ +import { BaseDeleteFilesRequest } from 'skyflow-common'; +import { DeleteFilesRecord } from '../../../types'; + +/** Request to delete files from one or more file columns. */ +class DeleteFilesRequest extends BaseDeleteFilesRequest { + private _records: DeleteFilesRecord[]; + + constructor(records: DeleteFilesRecord[]) { + super(); + this._records = records; + } + + public get records(): DeleteFilesRecord[] { + return this._records; + } + + public set records(value: DeleteFilesRecord[]) { + this._records = value; + } +} + +export default DeleteFilesRequest; diff --git a/flowvault/src/model/request/delete/index.ts b/flowvault/src/model/request/delete/index.ts new file mode 100644 index 00000000..0748dcba --- /dev/null +++ b/flowvault/src/model/request/delete/index.ts @@ -0,0 +1,43 @@ +import { BaseDeleteRequest } from 'skyflow-common'; + +/** Request to delete records by Skyflow ID or by unique column values. */ +class DeleteRequest extends BaseDeleteRequest { + private _table: string; + private _ids: string[]; + // A list of { columnName: value, ... } maps (mirrors Java/Python flowvault); + // each is sent on the wire as { data: {...} }. + private _uniqueValues?: Record[]; + + constructor(table: string, ids: string[], uniqueValues?: Record[]) { + super(); + this._table = table; + this._ids = ids; + this._uniqueValues = uniqueValues; + } + + public get table(): string { + return this._table; + } + + public set table(value: string) { + this._table = value; + } + + public get ids(): string[] { + return this._ids; + } + + public set ids(value: string[]) { + this._ids = value; + } + + public get uniqueValues(): Record[] | undefined { + return this._uniqueValues; + } + + public set uniqueValues(value: Record[] | undefined) { + this._uniqueValues = value; + } +} + +export default DeleteRequest; diff --git a/flowvault/src/model/request/detokenize/index.ts b/flowvault/src/model/request/detokenize/index.ts new file mode 100644 index 00000000..347479e3 --- /dev/null +++ b/flowvault/src/model/request/detokenize/index.ts @@ -0,0 +1,32 @@ +import { BaseDetokenizeRequest } from 'skyflow-common'; +import { TokenGroupRedaction } from '../../../types'; + +/** Request to detokenize a list of tokens. */ +class DetokenizeRequest extends BaseDetokenizeRequest { + private _tokens: string[]; + private _tokenGroupRedactions?: TokenGroupRedaction[]; + + constructor(tokens: string[], tokenGroupRedactions?: TokenGroupRedaction[]) { + super(); + this._tokens = tokens; + this._tokenGroupRedactions = tokenGroupRedactions; + } + + public get tokens(): string[] { + return this._tokens; + } + + public set tokens(value: string[]) { + this._tokens = value; + } + + public get tokenGroupRedactions(): TokenGroupRedaction[] | undefined { + return this._tokenGroupRedactions; + } + + public set tokenGroupRedactions(value: TokenGroupRedaction[] | undefined) { + this._tokenGroupRedactions = value; + } +} + +export default DetokenizeRequest; diff --git a/flowvault/src/model/request/get-tokens/index.ts b/flowvault/src/model/request/get-tokens/index.ts new file mode 100644 index 00000000..a603057d --- /dev/null +++ b/flowvault/src/model/request/get-tokens/index.ts @@ -0,0 +1,22 @@ +import { BaseGetTokensRequest } from 'skyflow-common'; +import { GetTokensRecord } from '../../../types'; + +/** Request to fetch deterministic tokens for a list of values. */ +class GetTokensRequest extends BaseGetTokensRequest { + private _records: GetTokensRecord[]; + + constructor(records: GetTokensRecord[]) { + super(); + this._records = records; + } + + public get records(): GetTokensRecord[] { + return this._records; + } + + public set records(value: GetTokensRecord[]) { + this._records = value; + } +} + +export default GetTokensRequest; diff --git a/flowvault/src/model/request/get/index.ts b/flowvault/src/model/request/get/index.ts new file mode 100644 index 00000000..8791e4e9 --- /dev/null +++ b/flowvault/src/model/request/get/index.ts @@ -0,0 +1,52 @@ +import { BaseGetRequest } from 'skyflow-common'; +import { ColumnRedaction } from '../../../types'; + +/** A single table/ids entry for a multi-table get request, with optional + * per-record column selection / redaction / unique-value lookup. */ +export interface GetRecord { + tableName: string; + ids: string[]; + columns?: string[]; + columnRedactions?: ColumnRedaction[]; + uniqueValues?: Record[]; +} + +/** Request to fetch records by Skyflow ID, optionally across tables. */ +class GetRequest extends BaseGetRequest { + private _table: string; + private _ids: string[]; + private _records?: GetRecord[]; + + constructor(table: string, ids: string[], records?: GetRecord[]) { + super(); + this._table = table; + this._ids = ids; + this._records = records; + } + + public get table(): string { + return this._table; + } + + public set table(value: string) { + this._table = value; + } + + public get ids(): string[] { + return this._ids; + } + + public set ids(value: string[]) { + this._ids = value; + } + + public get records(): GetRecord[] | undefined { + return this._records; + } + + public set records(value: GetRecord[] | undefined) { + this._records = value; + } +} + +export default GetRequest; diff --git a/flowvault/src/model/request/insert/index.ts b/flowvault/src/model/request/insert/index.ts new file mode 100644 index 00000000..217c2bd3 --- /dev/null +++ b/flowvault/src/model/request/insert/index.ts @@ -0,0 +1,32 @@ +import { BaseInsertRequest } from 'skyflow-common'; +import { InsertRecord } from '../../../types'; + +/** Request to insert one or more records into a table. */ +class InsertRequest extends BaseInsertRequest { + private _tableName: string; + private _records: InsertRecord[]; + + constructor(tableName: string, records: InsertRecord[]) { + super(); + this._tableName = tableName; + this._records = records; + } + + public get tableName(): string { + return this._tableName; + } + + public set tableName(value: string) { + this._tableName = value; + } + + public get records(): InsertRecord[] { + return this._records; + } + + public set records(value: InsertRecord[]) { + this._records = value; + } +} + +export default InsertRequest; diff --git a/flowvault/src/model/request/query/index.ts b/flowvault/src/model/request/query/index.ts new file mode 100644 index 00000000..366ce659 --- /dev/null +++ b/flowvault/src/model/request/query/index.ts @@ -0,0 +1,21 @@ +import { BaseQueryRequest } from 'skyflow-common'; + +/** Request to run a SQL-like query against a vault. */ +class QueryRequest extends BaseQueryRequest { + private _query: string; + + constructor(query: string) { + super(); + this._query = query; + } + + public get query(): string { + return this._query; + } + + public set query(value: string) { + this._query = value; + } +} + +export default QueryRequest; diff --git a/flowvault/src/model/request/update/index.ts b/flowvault/src/model/request/update/index.ts new file mode 100644 index 00000000..75191e1f --- /dev/null +++ b/flowvault/src/model/request/update/index.ts @@ -0,0 +1,43 @@ +import { BaseUpdateRequest } from 'skyflow-common'; +import { UpdateRecord } from '../../../types'; + +/** Request to update one or more records in a table. */ +class UpdateRequest extends BaseUpdateRequest { + private _tableName: string; + private _records: UpdateRecord[]; + // Request-level update type (e.g. "UPDATE" | "REPLACE"); mirrors Java/Python flowvault. + private _updateType?: string; + + constructor(tableName: string, records: UpdateRecord[], updateType?: string) { + super(); + this._tableName = tableName; + this._records = records; + this._updateType = updateType; + } + + public get tableName(): string { + return this._tableName; + } + + public set tableName(value: string) { + this._tableName = value; + } + + public get records(): UpdateRecord[] { + return this._records; + } + + public set records(value: UpdateRecord[]) { + this._records = value; + } + + public get updateType(): string | undefined { + return this._updateType; + } + + public set updateType(value: string | undefined) { + this._updateType = value; + } +} + +export default UpdateRequest; diff --git a/flowvault/src/model/request/upload-files/index.ts b/flowvault/src/model/request/upload-files/index.ts new file mode 100644 index 00000000..1c7ef273 --- /dev/null +++ b/flowvault/src/model/request/upload-files/index.ts @@ -0,0 +1,22 @@ +import { BaseUploadFilesRequest } from 'skyflow-common'; +import { UploadFilesRecord } from '../../../types'; + +/** Request to upload files into one or more file columns. */ +class UploadFilesRequest extends BaseUploadFilesRequest { + private _records: UploadFilesRecord[]; + + constructor(records: UploadFilesRecord[]) { + super(); + this._records = records; + } + + public get records(): UploadFilesRecord[] { + return this._records; + } + + public set records(value: UploadFilesRecord[]) { + this._records = value; + } +} + +export default UploadFilesRequest; diff --git a/flowvault/src/model/response/delete-files/index.ts b/flowvault/src/model/response/delete-files/index.ts new file mode 100644 index 00000000..58606ab5 --- /dev/null +++ b/flowvault/src/model/response/delete-files/index.ts @@ -0,0 +1,14 @@ +import { BaseDeleteFilesResponse } from 'skyflow-common'; +import { DeleteFilesResponseRecord } from '../../../types'; + +/** Response from a deleteFiles operation. */ +class DeleteFilesResponse extends BaseDeleteFilesResponse { + readonly records: DeleteFilesResponseRecord[]; + + constructor({ records }: { records: DeleteFilesResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default DeleteFilesResponse; diff --git a/flowvault/src/model/response/delete/index.ts b/flowvault/src/model/response/delete/index.ts new file mode 100644 index 00000000..38e8c90d --- /dev/null +++ b/flowvault/src/model/response/delete/index.ts @@ -0,0 +1,14 @@ +import { BaseDeleteResponse } from 'skyflow-common'; +import { DeleteResponseRecord } from '../../../types'; + +/** Response from a delete operation. */ +class DeleteResponse extends BaseDeleteResponse { + readonly records: DeleteResponseRecord[]; + + constructor({ records }: { records: DeleteResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default DeleteResponse; diff --git a/flowvault/src/model/response/detokenize/index.ts b/flowvault/src/model/response/detokenize/index.ts new file mode 100644 index 00000000..1ba9cb6c --- /dev/null +++ b/flowvault/src/model/response/detokenize/index.ts @@ -0,0 +1,14 @@ +import { BaseDetokenizeResponse } from 'skyflow-common'; +import { DetokenizeResponseRecord } from '../../../types'; + +/** Response from a detokenize operation. */ +class DetokenizeResponse extends BaseDetokenizeResponse { + readonly records: DetokenizeResponseRecord[]; + + constructor({ records }: { records: DetokenizeResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default DetokenizeResponse; diff --git a/flowvault/src/model/response/get-tokens/index.ts b/flowvault/src/model/response/get-tokens/index.ts new file mode 100644 index 00000000..d77b7f94 --- /dev/null +++ b/flowvault/src/model/response/get-tokens/index.ts @@ -0,0 +1,14 @@ +import { BaseGetTokensResponse } from 'skyflow-common'; +import { GetTokensResponseRecord } from '../../../types'; + +/** Response from a getTokens operation. */ +class GetTokensResponse extends BaseGetTokensResponse { + readonly records: GetTokensResponseRecord[]; + + constructor({ records }: { records: GetTokensResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default GetTokensResponse; diff --git a/flowvault/src/model/response/get/index.ts b/flowvault/src/model/response/get/index.ts new file mode 100644 index 00000000..de17c27d --- /dev/null +++ b/flowvault/src/model/response/get/index.ts @@ -0,0 +1,14 @@ +import { BaseGetResponse } from 'skyflow-common'; +import { GetResponseRecord } from '../../../types'; + +/** Response from a get operation. */ +class GetResponse extends BaseGetResponse { + readonly records: GetResponseRecord[]; + + constructor({ records }: { records: GetResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default GetResponse; diff --git a/flowvault/src/model/response/insert/index.ts b/flowvault/src/model/response/insert/index.ts new file mode 100644 index 00000000..ba146ff7 --- /dev/null +++ b/flowvault/src/model/response/insert/index.ts @@ -0,0 +1,14 @@ +import { BaseInsertResponse } from 'skyflow-common'; +import { InsertResponseRecord } from '../../../types'; + +/** Response from an insert operation. */ +class InsertResponse extends BaseInsertResponse { + readonly records: InsertResponseRecord[]; + + constructor({ records }: { records: InsertResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default InsertResponse; diff --git a/flowvault/src/model/response/query/index.ts b/flowvault/src/model/response/query/index.ts new file mode 100644 index 00000000..503efbf5 --- /dev/null +++ b/flowvault/src/model/response/query/index.ts @@ -0,0 +1,30 @@ +import { BaseQueryResponse } from 'skyflow-common'; +import { FlowVaultRecordError, QueryResponseMetadata } from '../../../types'; + +/** Response from a query operation. */ +class QueryResponse extends BaseQueryResponse { + readonly fields: Record[]; + readonly errors: FlowVaultRecordError[] | null; + readonly metadata: QueryResponseMetadata; + readonly requestId: string; + + constructor({ + fields, + errors, + metadata, + requestId, + }: { + fields: Record[]; + errors: FlowVaultRecordError[] | null; + metadata: QueryResponseMetadata; + requestId: string; + }) { + super(); + this.fields = fields; + this.errors = errors; + this.metadata = metadata; + this.requestId = requestId; + } +} + +export default QueryResponse; diff --git a/flowvault/src/model/response/update/index.ts b/flowvault/src/model/response/update/index.ts new file mode 100644 index 00000000..9a2867e6 --- /dev/null +++ b/flowvault/src/model/response/update/index.ts @@ -0,0 +1,14 @@ +import { BaseUpdateResponse } from 'skyflow-common'; +import { UpdateResponseRecord } from '../../../types'; + +/** Response from an update operation. */ +class UpdateResponse extends BaseUpdateResponse { + readonly records: UpdateResponseRecord[]; + + constructor({ records }: { records: UpdateResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default UpdateResponse; diff --git a/flowvault/src/model/response/upload-files/index.ts b/flowvault/src/model/response/upload-files/index.ts new file mode 100644 index 00000000..ddb9e089 --- /dev/null +++ b/flowvault/src/model/response/upload-files/index.ts @@ -0,0 +1,14 @@ +import { BaseUploadFilesResponse } from 'skyflow-common'; +import { UploadFilesResponseRecord } from '../../../types'; + +/** Response from an uploadFiles operation. */ +class UploadFilesResponse extends BaseUploadFilesResponse { + readonly records: UploadFilesResponseRecord[]; + + constructor({ records }: { records: UploadFilesResponseRecord[] }) { + super(); + this.records = records; + } +} + +export default UploadFilesResponse; diff --git a/flowvault/src/skyflow/index.ts b/flowvault/src/skyflow/index.ts new file mode 100644 index 00000000..244f7c8d --- /dev/null +++ b/flowvault/src/skyflow/index.ts @@ -0,0 +1,73 @@ +import { + BaseSkyflow, + BaseVaultEntry, + Env, + MessageType, + SkyflowConfig, + VaultConfig, + logs, + printLog, + setSdkIdentity, +} from 'skyflow-common'; +import VaultClient from '../client'; +import VaultController from '../controller/vault'; +import { SDK_NAME, SDK_VERSION } from '../version'; + +// Stamp the SDK identity so telemetry and errors self-report FlowVault. +setSdkIdentity({ name: SDK_NAME, version: SDK_VERSION }); + +// FlowVault (v3) is hosted on the `skyvault.*` subdomain, a DIFFERENT host than +// the shared `getVaultURL` (`vault.*`, used by skyvault). All four confirmed. +function getFlowVaultURL(clusterId: string, env: Env): string { + switch (env) { + case Env.PROD: + return `https://${clusterId}.skyvault.skyflowapis.com`; + case Env.SANDBOX: + return `https://${clusterId}.skyvault.skyflowapis-preview.com`; + case Env.DEV: + return `https://${clusterId}.skyvault.skyflowapis.dev`; + case Env.STAGE: + return `https://${clusterId}.skyvault.skyflowapis.tech`; + default: + return `https://${clusterId}.skyvault.skyflowapis.com`; + } +} + +interface ClientEntry extends BaseVaultEntry { + vaultController: VaultController; +} + +/** + * Skyflow is the FlowVault SDK facade. Vault-config / credential / log-level + * management comes from the shared `BaseSkyflow` in `common`; this layer only + * builds its own VaultClient + VaultController and exposes `vault(vaultId?)`. + * (FlowVault has no connections or detect.) + */ +class Skyflow extends BaseSkyflow { + constructor(config: SkyflowConfig) { + super(); + this.initializeSkyflow(config); + printLog(logs.infoLogs.CLIENT_INITIALIZED, MessageType.LOG, this.logLevel); + } + + protected buildVaultEntry(config: VaultConfig): ClientEntry { + const env = config.env || Env.PROD; + const vaultUrl = config.vaultUrl || getFlowVaultURL(config.clusterId, env); + const client = new VaultClient( + vaultUrl, + config.vaultId, + config?.credentials, + this.commonCredentials, + this.logLevel, + { timeout: config.timeout, maxRetries: config.maxRetries }, + ); + const vaultController = new VaultController(client); + return { config, client, vaultController }; + } + + vault(vaultId?: string): VaultController { + return this.resolveVaultEntry(vaultId).vaultController; + } +} + +export default Skyflow; diff --git a/flowvault/src/types/index.ts b/flowvault/src/types/index.ts new file mode 100644 index 00000000..dc27da08 --- /dev/null +++ b/flowvault/src/types/index.ts @@ -0,0 +1,187 @@ +// Shared FlowVault types: record shapes for requests and the unified +// response records. Response envelopes themselves are the Response classes +// under `model/response`. + +import { SkyflowRecordError } from 'skyflow-common'; + +/** A single record to insert. `tokens` enables BYOT (bring your own token). + * `table` is an optional per-record table name — when any record sets it, the + * records can target multiple tables in a single insert call (matches the + * Java/Python flowvault SDKs). */ +export interface InsertRecord { + data: Record; + tokens?: Record; + table?: string; + /** Per-record upsert; when any record sets it, request-level upsert is ignored. */ + upsert?: UpsertOptions; +} + +/** Upsert configuration for insert operations. */ +export interface UpsertOptions { + updateType: string; + uniqueColumns: string[]; +} + +/** Column level redaction used by get. */ +export interface ColumnRedaction { + columnName: string; + redaction: string; +} + +/** A single record to update. `table` is an optional per-record table name + * (lets a single update call target multiple tables, like insert). */ +export interface UpdateRecord { + skyflowId: string; + data: Record; + tokens?: Record; + table?: string; +} + +/** Token group redaction used by detokenize. */ +export interface TokenGroupRedaction { + tokenGroupName: string; + redaction: string; +} + +/** A value / token-group pair to tokenize via getTokens. */ +export interface GetTokensRecord { + value: unknown; + tokenGroupName: string; +} + +/** A single file column to upload. Provide exactly one of filePath/base64/file. */ +export interface UploadFilesColumn { + column: string; + filePath?: string; + base64?: string; + file?: File; + fileName?: string; +} + +/** A record (row) with one or more file columns to upload. */ +export interface UploadFilesRecord { + tableName: string; + skyflowId?: string; + columns: UploadFilesColumn[]; +} + +/** A record (row) with one or more file columns to delete. */ +export interface DeleteFilesRecord { + tableName: string; + columns: string[]; + skyflowId?: string; + uniqueValues?: Record[]; +} + +/** Result of uploading a file to a single column. */ +export type UploadStatus = 'UPLOADED' | 'FAILED' | 'SKIPPED'; + +// --------------------------------------------------------------------------- +// Unified response records. Every record carries httpCode and an optional +// error. requestId is populated only when error is present. +// --------------------------------------------------------------------------- + +export interface InsertResponseRecord { + tableName?: string | null; + skyflowId?: string | null; + // tokens and data are surfaced separately (matches the Java/Python flowvault + // SDKs); either may be present depending on the vault configuration. + tokens?: Record | null; + data?: Record | null; + hashedData?: Record | null; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +export interface GetResponseRecord { + skyflowId?: string | null; + tableName?: string | null; + data?: Record | null; + hashedData?: Record | null; + tokens?: Record | null; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +export interface UpdateResponseRecord { + skyflowId?: string | null; + tableName?: string | null; + tokens?: Record | null; + data?: Record | null; + hashedData?: Record | null; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +// NOTE: the delete response record intentionally has NO tableName — the v2 +// delete API (and the Java/Python flowvault SDKs) return only skyflowId. +export interface DeleteResponseRecord { + skyflowId?: string | null; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +export interface DetokenizeResponseRecord { + token: string; + tokenGroupName?: string | null; + value?: unknown; + metadata?: { skyflowId?: string | null; tableName?: string | null } | null; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +export interface GetTokensResponseRecord { + value: unknown; + tokenGroupName: string | null; + token?: string | null; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +export interface UploadFilesResponseColumn { + column: string; + fileName?: string | null; + uploadStatus: UploadStatus; + error?: string | null; +} + +export interface UploadFilesResponseRecord { + skyflowId?: string | null; + tableName?: string | null; + columns: UploadFilesResponseColumn[]; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +export interface DeleteFilesResponseColumn { + column: string; + // Status is parsed from the API response value (e.g. "DELETED"), not assumed. + status: string | null; +} + +export interface DeleteFilesResponseRecord { + skyflowId?: string | null; + tableName?: string | null; + columns: DeleteFilesResponseColumn[] | null; + httpCode: number; + error?: string | null; + requestId?: string | null; +} + +/** Metadata block returned by query. */ +export interface QueryResponseMetadata { + columns: string[]; +} + +/** Record-level error shape re-used by query. */ +export type FlowVaultRecordError = SkyflowRecordError; + +/** Internal: registry entry for a configured vault. */ +export type { SkyflowConfig } from 'skyflow-common'; diff --git a/flowvault/src/validations/index.ts b/flowvault/src/validations/index.ts new file mode 100644 index 00000000..71fcd9bd --- /dev/null +++ b/flowvault/src/validations/index.ts @@ -0,0 +1,213 @@ +import { + SkyflowError, + SKYFLOW_ERROR_CODE, + LogLevel, + MessageType, + printLog, +} from 'skyflow-common'; +import InsertRequest from '../model/request/insert'; +import GetRequest from '../model/request/get'; +import GetOptions from '../model/options/get'; +import UpdateRequest from '../model/request/update'; +import DeleteRequest from '../model/request/delete'; +import QueryRequest from '../model/request/query'; +import DetokenizeRequest from '../model/request/detokenize'; +import GetTokensRequest from '../model/request/get-tokens'; +import UploadFilesRequest from '../model/request/upload-files'; +import DeleteFilesRequest from '../model/request/delete-files'; + +const isNonEmptyString = (value: unknown): boolean => + typeof value === 'string' && value.trim().length > 0; + +const isNonEmptyArray = (value: unknown): boolean => + Array.isArray(value) && value.length > 0; + +const fail = (code: { httpCode: number; message: string }, logLevel?: LogLevel): never => { + printLog(code.message, MessageType.ERROR, logLevel); + throw new SkyflowError(code); +}; + +export const validateInsertRequest = ( + request: InsertRequest, + _options?: unknown, + logLevel?: LogLevel, +): void => { + if (!request || !(request instanceof InsertRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_INSERT_REQUEST, logLevel); + } + if (!isNonEmptyString(request.tableName)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME, logLevel); + } + if (!isNonEmptyArray(request.records)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_RECORD_IN_INSERT, logLevel); + } + request.records.forEach((record) => { + if (!record || typeof record.data !== 'object' || record.data === null) { + fail(SKYFLOW_ERROR_CODE.EMPTY_DATA_IN_INSERT, logLevel); + } + }); +}; + +export const validateGetRequest = ( + request: GetRequest, + options?: GetOptions, + logLevel?: LogLevel, +): void => { + if (!request || !(request instanceof GetRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_GET_REQUEST, logLevel); + } + const hasMultiTable = isNonEmptyArray(request.records); + if (!hasMultiTable) { + if (!isNonEmptyString(request.table)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME, logLevel); + } + // Single-table lookup is by ids OR by unique values (unique values live on + // GetOptions); require at least one of them. + const hasUniqueValues = isNonEmptyArray(options?.getUniqueValues()); + if (!isNonEmptyArray(request.ids) && !hasUniqueValues) { + fail(SKYFLOW_ERROR_CODE.EMPTY_IDS_IN_GET, logLevel); + } + } +}; + +export const validateUpdateRequest = ( + request: UpdateRequest, + _options?: unknown, + logLevel?: LogLevel, +): void => { + if (!request || !(request instanceof UpdateRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_UPDATE_REQUEST, logLevel); + } + if (!isNonEmptyString(request.tableName)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME, logLevel); + } + if (!isNonEmptyArray(request.records)) { + fail(SKYFLOW_ERROR_CODE.INVALID_RECORD_IN_UPDATE, logLevel); + } + request.records.forEach((record) => { + if (!isNonEmptyString(record?.skyflowId)) { + fail(SKYFLOW_ERROR_CODE.MISSING_SKYFLOW_ID_IN_UPDATE, logLevel); + } + if (!record || typeof record.data !== 'object' || record.data === null) { + fail(SKYFLOW_ERROR_CODE.EMPTY_DATA_IN_UPDATE, logLevel); + } + }); +}; + +export const validateDeleteRequest = (request: DeleteRequest, logLevel?: LogLevel): void => { + if (!request || !(request instanceof DeleteRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_DELETE_REQUEST, logLevel); + } + if (!isNonEmptyString(request.table)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME, logLevel); + } + const hasUniqueValues = isNonEmptyArray(request.uniqueValues); + if (!hasUniqueValues && !isNonEmptyArray(request.ids)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_DELETE_IDS, logLevel); + } +}; + +export const validateQueryRequest = (request: QueryRequest, logLevel?: LogLevel): void => { + if (!request || !(request instanceof QueryRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_QUERY_REQUEST, logLevel); + } + if (!isNonEmptyString(request.query)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_QUERY, logLevel); + } +}; + +export const validateDetokenizeRequest = ( + request: DetokenizeRequest, + _options?: unknown, + logLevel?: LogLevel, +): void => { + if (!request || !(request instanceof DetokenizeRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_DETOKENIZE_REQUEST, logLevel); + } + if (!isNonEmptyArray(request.tokens)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_TOKENS_IN_DETOKENIZE, logLevel); + } + request.tokens.forEach((token) => { + if (!isNonEmptyString(token)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_TOKEN_IN_DETOKENIZE, logLevel); + } + }); +}; + +export const validateGetTokensRequest = ( + request: GetTokensRequest, + logLevel?: LogLevel, +): void => { + if (!request || !(request instanceof GetTokensRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_TOKENIZE_REQUEST, logLevel); + } + if (!isNonEmptyArray(request.records)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_VALUES_IN_TOKENIZE, logLevel); + } + request.records.forEach((record) => { + if (!isNonEmptyString(record?.tokenGroupName)) { + fail(SKYFLOW_ERROR_CODE.EMPTY_COLUMN_GROUP_IN_TOKENIZE, logLevel); + } + }); +}; + +export const validateUploadFilesRequest = ( + request: UploadFilesRequest, + _options?: unknown, + logLevel?: LogLevel, +): void => { + if (!request || !(request instanceof UploadFilesRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST, logLevel); + } + if (!isNonEmptyArray(request.records)) { + fail(SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST, logLevel); + } + request.records.forEach((record) => { + if (!isNonEmptyString(record?.tableName)) { + fail(SKYFLOW_ERROR_CODE.MISSING_TABLE_IN_UPLOAD_FILE, logLevel); + } + if (!isNonEmptyArray(record?.columns)) { + fail(SKYFLOW_ERROR_CODE.MISSING_COLUMN_NAME_IN_UPLOAD_FILE, logLevel); + } + record.columns.forEach((column) => { + if (!isNonEmptyString(column?.column)) { + fail(SKYFLOW_ERROR_CODE.MISSING_COLUMN_NAME_IN_UPLOAD_FILE, logLevel); + } + const sources = [column.filePath, column.base64, column.file].filter( + (source) => source !== undefined && source !== null, + ); + if (sources.length !== 1) { + fail(SKYFLOW_ERROR_CODE.MISSING_FILE_SOURCE_IN_UPLOAD_FILE, logLevel); + } + if (column.base64 !== undefined && !isNonEmptyString(column.fileName)) { + fail(SKYFLOW_ERROR_CODE.MISSING_FILE_NAME_FOR_BASE64, logLevel); + } + }); + }); +}; + +export const validateDeleteFilesRequest = ( + request: DeleteFilesRequest, + logLevel?: LogLevel, +): void => { + if (!request || !(request instanceof DeleteFilesRequest)) { + fail(SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST, logLevel); + } + if (!isNonEmptyArray(request.records)) { + fail(SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST, logLevel); + } + request.records.forEach((record) => { + if (!isNonEmptyString(record?.tableName)) { + fail(SKYFLOW_ERROR_CODE.MISSING_TABLE_IN_UPLOAD_FILE, logLevel); + } + if (!isNonEmptyArray(record?.columns)) { + fail(SKYFLOW_ERROR_CODE.MISSING_COLUMN_NAME_IN_UPLOAD_FILE, logLevel); + } + const hasSkyflowId = isNonEmptyString(record?.skyflowId); + const hasUniqueValues = isNonEmptyArray(record?.uniqueValues); + if (hasSkyflowId === hasUniqueValues) { + // exactly one of skyflowId / uniqueValues is required + fail(SKYFLOW_ERROR_CODE.MISSING_SKYFLOW_ID_IN_UPLOAD_FILE, logLevel); + } + }); +}; diff --git a/flowvault/src/version.ts b/flowvault/src/version.ts new file mode 100644 index 00000000..48308455 --- /dev/null +++ b/flowvault/src/version.ts @@ -0,0 +1,4 @@ +// SDK version constant for skyflow-flowvault-node. +// Kept in sync with flowvault/package.json "version" by scripts/bump_version.sh. +export const SDK_NAME = "skyflow-flowvault-node"; +export const SDK_VERSION = "1.0.0"; diff --git a/flowvault/test/client/client.test.ts b/flowvault/test/client/client.test.ts new file mode 100644 index 00000000..7290a44c --- /dev/null +++ b/flowvault/test/client/client.test.ts @@ -0,0 +1,205 @@ +// Mock only getBearerToken from skyflow-common; everything else stays real so +// failureResponse / getCredentials / isExpired behave as in production. +jest.mock('skyflow-common', () => { + const actual = jest.requireActual('skyflow-common'); + return { ...actual, getBearerToken: jest.fn() }; +}); + +import { + AuthType, + LogLevel, + SkyflowError, + SKYFLOW_ERROR_CODE, + errorMessages, + getBearerToken, +} from '../../src'; +import VaultClient from '../../src/client'; +import { makeJwt, silenceConsole } from '../helpers'; + +silenceConsole(); + +const getBearerTokenMock = getBearerToken as unknown as jest.Mock; + +const URL = 'https://abc-cluster.vault.skyflowapis.com'; + +function newClient(individual?: unknown, common?: unknown): VaultClient { + return new VaultClient(URL, 'vault123', individual as never, common as never, LogLevel.ERROR); +} + +describe('VaultClient.failureResponse', () => { + it('normalizes a snake_case v2 error envelope; httpCode from statusCode, requestId from header', () => { + const client = newClient({ apiKey: 'sky-x' }); + const err = { + statusCode: 404, + body: { error: { grpc_code: 5, http_code: 404, http_status: 'Not Found', message: 'not found', details: ['d1'] } }, + rawResponse: { headers: new Map([['x-request-id', 'req-xyz']]) }, + }; + + const result = client.failureResponse(err); + + expect(result).toBeInstanceOf(SkyflowError); + expect(result.error?.httpCode).toBe(404); // from statusCode (takes priority) + expect(result.error?.grpcCode).toBe(5); + expect(result.error?.httpStatus).toBe('Not Found'); + expect(result.error?.message).toBe('not found'); + expect(result.error?.details).toEqual(['d1']); + expect(result.error?.requestId).toBe('req-xyz'); // from header (no body requestId) + }); + + it('normalizes a camelCase variant; requestId comes from the body when present', () => { + const client = newClient({ apiKey: 'sky-x' }); + const err = { + statusCode: 400, + body: { error: { grpcCode: 3, httpCode: 400, httpStatus: 'Bad Request', message: 'bad', details: [], requestId: 'rid-1' } }, + }; + + const result = client.failureResponse(err); + + expect(result.error?.httpCode).toBe(400); + expect(result.error?.grpcCode).toBe(3); + expect(result.error?.httpStatus).toBe('Bad Request'); + expect(result.error?.message).toBe('bad'); + expect(result.error?.requestId).toBe('rid-1'); + }); + + it('falls back to err.message and defaults when there is no body', () => { + const client = newClient({ apiKey: 'sky-x' }); + const result = client.failureResponse({ message: 'boom' }); + + expect(result.error?.message).toBe('boom'); + expect(result.error?.httpCode).toBe(400); // HTTP_STATUS_CODE.BAD_REQUEST default + expect(result.error?.grpcCode).toBeNull(); + expect(result.error?.httpStatus).toBe('Bad Request'); + expect(result.error?.requestId).toBeNull(); + expect(result.error?.details).toEqual([]); + }); + + it('uses the generic ERROR_OCCURRED message when nothing else is present', () => { + const client = newClient({ apiKey: 'sky-x' }); + const result = client.failureResponse({}); + expect(result.error?.message).toBe(errorMessages.ERROR_OCCURRED); + }); + + it('passes an existing SkyflowError straight through', () => { + const client = newClient({ apiKey: 'sky-x' }); + const original = new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_QUERY); + expect(client.failureResponse(original)).toBe(original); + }); +}); + +describe('VaultClient.getToken', () => { + beforeEach(() => getBearerTokenMock.mockReset()); + + it('resolves a token via getBearerToken and wires the resource clients to token/baseUrl suppliers', async () => { + getBearerTokenMock.mockResolvedValue({ key: 'tok', type: AuthType.TOKEN }); + const client = newClient({ apiKey: 'sky-x' }); + + await client.getToken(); + + expect(getBearerTokenMock).toHaveBeenCalledTimes(1); + // Each generated resource client gets baseUrl + token suppliers. + for (const api of [client.records, client.tokens, client.files, client.query]) { + const opts = (api as unknown as { _options: { baseUrl: () => string; token: () => string } })._options; + expect(opts.baseUrl()).toBe(URL); + expect(opts.token()).toBe('tok'); + } + }); +}); + +describe('VaultClient.getCredentials', () => { + it('reuses an API key from authInfo without re-reading credentials', () => { + const client = newClient({ apiKey: 'sky-individual' }); + (client as unknown as { authInfo: unknown }).authInfo = { key: 'sky-cached', type: AuthType.API_KEY }; + expect(client.getCredentials()).toEqual({ apiKey: 'sky-cached' }); + }); + + it('reuses a non-expired bearer token from authInfo', () => { + const client = newClient({ apiKey: 'sky-individual' }); + const validJwt = makeJwt(3600); + (client as unknown as { authInfo: unknown }).authInfo = { key: validJwt, type: AuthType.TOKEN }; + expect(client.getCredentials()).toEqual({ token: validJwt }); + }); + + it('falls back to individual credentials when the cached bearer token is expired', () => { + const client = newClient({ apiKey: 'sky-individual' }); + const expiredJwt = makeJwt(-3600); + (client as unknown as { authInfo: unknown }).authInfo = { key: expiredJwt, type: AuthType.TOKEN }; + expect(client.getCredentials()).toEqual({ apiKey: 'sky-individual' }); + }); + + it('returns individual over common when there is no cached authInfo', () => { + const client = newClient({ apiKey: 'sky-individual' }, { apiKey: 'sky-common' }); + expect(client.getCredentials()).toEqual({ apiKey: 'sky-individual' }); + }); + + it('returns common credentials when no individual credentials are set', () => { + const client = newClient(undefined, { apiKey: 'sky-common' }); + expect(client.getCredentials()).toEqual({ apiKey: 'sky-common' }); + }); + + it('falls through to stored credentials for an unrecognized cached auth type', () => { + // Defensive: an authInfo.type the switch does not handle hits the `default` + // arm and falls back to the configured credentials. + const client = newClient({ apiKey: 'sky-individual' }); + (client as unknown as { authInfo: unknown }).authInfo = { key: 'sky-cached', type: 'MYSTERY' as never }; + expect(client.getCredentials()).toEqual({ apiKey: 'sky-individual' }); + }); + + it('skips reuse right after updateSkyflowCredentials (updateTriggered), then resumes', () => { + const client = newClient({ apiKey: 'sky-individual' }); + (client as unknown as { authInfo: unknown }).authInfo = { key: 'sky-cached', type: AuthType.API_KEY }; + client.updateSkyflowCredentials({ apiKey: 'sky-common' } as never); + // updateTriggered -> ignore cached authInfo, return individual ?? common. + expect(client.getCredentials()).toEqual({ apiKey: 'sky-individual' }); + // Subsequent call reuses the cached key again (updateTriggered reset). + expect(client.getCredentials()).toEqual({ apiKey: 'sky-cached' }); + }); +}); + +describe('VaultClient config updates', () => { + it('updateClientConfig rebuilds resource clients against the new URL', async () => { + getBearerTokenMock.mockResolvedValue({ key: 'tok', type: AuthType.TOKEN }); + const client = newClient({ apiKey: 'sky-x' }); + await client.getToken(); + const before = client.records; + + const newUrl = 'https://new-cluster.vault.skyflowapis.com'; + client.updateClientConfig(newUrl, 'vault123', { apiKey: 'sky-y' } as never, undefined, LogLevel.DEBUG); + + const after = client.records; + expect(after).not.toBe(before); // rebuilt + expect(client.getLogLevel()).toBe(LogLevel.DEBUG); + const opts = (after as unknown as { _options: { baseUrl: () => string } })._options; + expect(opts.baseUrl()).toBe(newUrl); + }); + + it('setLogLevel updates the stored level', () => { + const client = newClient({ apiKey: 'sky-x' }); + client.setLogLevel(LogLevel.WARN); + expect(client.getLogLevel()).toBe(LogLevel.WARN); + }); + + it('defaults the log level to ERROR when the constructor gets none', () => { + const client = new VaultClient(URL, 'vault123', { apiKey: 'sky-x' } as never); + expect(client.getLogLevel()).toBe(LogLevel.ERROR); + }); + + it('stores timeout and maxRetries from httpConfig (undefined when omitted)', () => { + const configured = new VaultClient(URL, 'vault123', { apiKey: 'sky-x' } as never, undefined, LogLevel.ERROR, { + timeout: 45, + maxRetries: 3, + }); + expect(configured.timeout).toBe(45); + expect(configured.maxRetries).toBe(3); + + const unconfigured = new VaultClient(URL, 'vault123', { apiKey: 'sky-x' } as never); + expect(unconfigured.timeout).toBeUndefined(); + expect(unconfigured.maxRetries).toBeUndefined(); + }); + + it('updateClientConfig defaults the log level to ERROR when none is provided', () => { + const client = newClient({ apiKey: 'sky-x' }); + client.updateClientConfig(URL, 'vault123', { apiKey: 'sky-y' } as never, undefined, undefined); + expect(client.getLogLevel()).toBe(LogLevel.ERROR); + }); +}); diff --git a/flowvault/test/controller/delete-files.test.ts b/flowvault/test/controller/delete-files.test.ts new file mode 100644 index 00000000..13811d35 --- /dev/null +++ b/flowvault/test/controller/delete-files.test.ts @@ -0,0 +1,122 @@ +import { DeleteFilesRequest, DeleteFilesResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoHeaders, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +describe('VaultController.deleteFiles', () => { + it('builds skyflowID + columns and parses status (object->.status, else value)', async () => { + const deleteFiles = resourceMethod({ + records: [ + { + skyflowID: 'id1', + tableName: 'onboarding', + data: { resume: { status: 'DELETED' }, photo: 'DELETED' }, + httpCode: 200, + }, + ], + }); + const controller = new VaultController(makeFakeClient({ files: { deleteFiles } })); + + const req = new DeleteFilesRequest([ + { tableName: 'onboarding', columns: ['resume', 'photo'], skyflowId: 'id1' }, + ]); + const resp = await controller.deleteFiles(req); + + expect(resp).toBeInstanceOf(DeleteFilesResponse); + expect(deleteFiles).toHaveBeenCalledTimes(1); + const [genReq] = deleteFiles.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.records[0]).toEqual({ tableName: 'onboarding', columns: ['resume', 'photo'], skyflowID: 'id1' }); + + const record = resp.records[0]; + expect(record.skyflowId).toBe('id1'); + expect(record.tableName).toBe('onboarding'); + expect(record.httpCode).toBe(200); + expect(record.error).toBeNull(); + expect(record.requestId).toBeUndefined(); + expect(record.columns).toEqual([ + { column: 'resume', status: 'DELETED' }, // object -> .status + { column: 'photo', status: 'DELETED' }, // scalar -> value + ]); + }); + + it('maps uniqueValues->{data} and omits skyflowID when absent', async () => { + const deleteFiles = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ files: { deleteFiles } })); + + await controller.deleteFiles( + new DeleteFilesRequest([ + { tableName: 't', columns: ['c'], uniqueValues: [{ email: 'a@b.com' }, { ssn: '111' }] }, + ]), + ); + + const [genReq] = deleteFiles.mock.calls[0]; + expect(genReq.records[0].skyflowID).toBeUndefined(); + expect(genReq.records[0].uniqueValues).toEqual([{ data: { email: 'a@b.com' } }, { data: { ssn: '111' } }]); + }); + + it('columns is null and requestId is set from header on a failed record (no data)', async () => { + const deleteFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', httpCode: 404, error: 'not found' }], + }); + const controller = new VaultController(makeFakeClient({ files: { deleteFiles } })); + + const resp = await controller.deleteFiles( + new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'id1' }]), + ); + + const record = resp.records[0]; + expect(record.columns).toBeNull(); + expect(record.error).toBe('not found'); + expect(record.requestId).toBe(REQUEST_ID); + }); + + it('parses a null column value as a null status', async () => { + const deleteFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { resume: null }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { deleteFiles } })); + const resp = await controller.deleteFiles( + new DeleteFilesRequest([{ tableName: 't', columns: ['resume'], skyflowId: 'id1' }]), + ); + expect(resp.records[0].columns).toEqual([{ column: 'resume', status: null }]); + }); + + it('returns an empty records array when the response has none', async () => { + const deleteFiles = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ files: { deleteFiles } })); + const resp = await controller.deleteFiles( + new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'id1' }]), + ); + expect(resp.records).toEqual([]); + }); + + it('sets requestId to null on an errored record when the raw response has no headers', async () => { + const deleteFiles = resourceMethod( + { records: [{ skyflowID: 'id1', tableName: 't', httpCode: 404, error: 'not found' }] }, + rawResponseNoHeaders, + ); + const controller = new VaultController(makeFakeClient({ files: { deleteFiles } })); + const resp = await controller.deleteFiles( + new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'id1' }]), + ); + expect(resp.records[0].error).toBe('not found'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('rejects when the generated method throws', async () => { + const deleteFiles = rejectingMethod(new Error('boom-delete-files')); + const controller = new VaultController(makeFakeClient({ files: { deleteFiles } })); + await expect( + controller.deleteFiles(new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'id1' }])), + ).rejects.toThrow('boom-delete-files'); + }); +}); diff --git a/flowvault/test/controller/delete.test.ts b/flowvault/test/controller/delete.test.ts new file mode 100644 index 00000000..613c7ac9 --- /dev/null +++ b/flowvault/test/controller/delete.test.ts @@ -0,0 +1,78 @@ +import { DeleteRequest, DeleteResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoHeaders, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +describe('VaultController.delete', () => { + it('builds skyflowIDs and maps the response (no tableName on record)', async () => { + const deleteRecords = resourceMethod({ + records: [{ skyflowID: 'id1', httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ records: { deleteRecords } })); + + const resp = await controller.delete(new DeleteRequest('cards', ['id1', 'id2'])); + + expect(resp).toBeInstanceOf(DeleteResponse); + expect(deleteRecords).toHaveBeenCalledTimes(1); + const [genReq] = deleteRecords.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.tableName).toBe('cards'); + expect(genReq.skyflowIDs).toEqual(['id1', 'id2']); + expect(genReq.uniqueValues).toBeUndefined(); + + const record = resp.records[0]; + expect(record.skyflowId).toBe('id1'); + expect(record.httpCode).toBe(200); + expect(record.error).toBeNull(); + expect(record.requestId).toBeUndefined(); + expect((record as Record).tableName).toBeUndefined(); + }); + + it('builds uniqueValues->{data} when ids are absent', async () => { + const deleteRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { deleteRecords } })); + + await controller.delete(new DeleteRequest('cards', [], [{ email: 'a@b.com' }, { ssn: '111' }])); + + const [genReq] = deleteRecords.mock.calls[0]; + expect(genReq.skyflowIDs).toBeUndefined(); + expect(genReq.uniqueValues).toEqual([{ data: { email: 'a@b.com' } }, { data: { ssn: '111' } }]); + }); + + it('per-record error gets requestId from the header', async () => { + const deleteRecords = resourceMethod({ records: [{ skyflowID: 'id1', httpCode: 404, error: 'missing' }] }); + const controller = new VaultController(makeFakeClient({ records: { deleteRecords } })); + const resp = await controller.delete(new DeleteRequest('cards', ['id1'])); + expect(resp.records[0].error).toBe('missing'); + expect(resp.records[0].requestId).toBe(REQUEST_ID); + }); + + it('returns an empty records array when the response has none', async () => { + const deleteRecords = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ records: { deleteRecords } })); + const resp = await controller.delete(new DeleteRequest('cards', ['id1'])); + expect(resp.records).toEqual([]); + }); + + it('sets requestId to null on an errored record when the raw response has no headers', async () => { + const deleteRecords = resourceMethod({ records: [{ skyflowID: 'id1', httpCode: 404, error: 'missing' }] }, rawResponseNoHeaders); + const controller = new VaultController(makeFakeClient({ records: { deleteRecords } })); + const resp = await controller.delete(new DeleteRequest('cards', ['id1'])); + expect(resp.records[0].error).toBe('missing'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('rejects when the generated method throws', async () => { + const deleteRecords = rejectingMethod(new Error('boom-delete')); + const controller = new VaultController(makeFakeClient({ records: { deleteRecords } })); + await expect(controller.delete(new DeleteRequest('cards', ['id1']))).rejects.toThrow('boom-delete'); + }); +}); diff --git a/flowvault/test/controller/detokenize.test.ts b/flowvault/test/controller/detokenize.test.ts new file mode 100644 index 00000000..e083a12f --- /dev/null +++ b/flowvault/test/controller/detokenize.test.ts @@ -0,0 +1,108 @@ +import { DetokenizeRequest, DetokenizeResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoHeaders, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +describe('VaultController.detokenize', () => { + it('reads the generated `response` array and maps records (metadata skyflowID->skyflowId)', async () => { + const detokenize = resourceMethod({ + response: [ + { + token: 'tok1', + value: '4111', + tokenGroupName: 'cards', + httpCode: 200, + metadata: { skyflowID: 'id1', tableName: 'cards' }, + }, + ], + }); + const controller = new VaultController(makeFakeClient({ tokens: { detokenize } })); + + const req = new DetokenizeRequest(['tok1'], [{ tokenGroupName: 'cards', redaction: 'PLAIN_TEXT' }]); + const resp = await controller.detokenize(req); + + expect(resp).toBeInstanceOf(DetokenizeResponse); + expect(detokenize).toHaveBeenCalledTimes(1); + const [genReq] = detokenize.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.tokens).toEqual(['tok1']); + expect(genReq.tokenGroupRedactions).toEqual([{ tokenGroupName: 'cards', redaction: 'PLAIN_TEXT' }]); + + const record = resp.records[0]; + expect(record.token).toBe('tok1'); + expect(record.value).toBe('4111'); + expect(record.tokenGroupName).toBe('cards'); + expect(record.metadata).toEqual({ skyflowId: 'id1', tableName: 'cards' }); + expect(record.httpCode).toBe(200); + expect(record.error).toBeNull(); + expect(record.requestId).toBeUndefined(); + }); + + it('defaults httpCode to 200, metadata to null, and omits redactions when absent', async () => { + const detokenize = resourceMethod({ + response: [{ token: 'tok1', value: '4111' }], + }); + const controller = new VaultController(makeFakeClient({ tokens: { detokenize } })); + + const resp = await controller.detokenize(new DetokenizeRequest(['tok1'])); + + const [genReq] = detokenize.mock.calls[0]; + expect(genReq.tokenGroupRedactions).toBeUndefined(); + + const record = resp.records[0]; + expect(record.httpCode).toBe(200); + expect(record.metadata).toBeNull(); + expect(record.tokenGroupName).toBeNull(); + }); + + it('accepts camelCase metadata.skyflowId as well', async () => { + const detokenize = resourceMethod({ + response: [{ token: 'tok1', value: 'v', metadata: { skyflowId: 'id9' } }], + }); + const controller = new VaultController(makeFakeClient({ tokens: { detokenize } })); + const resp = await controller.detokenize(new DetokenizeRequest(['tok1'])); + expect(resp.records[0].metadata).toEqual({ skyflowId: 'id9', tableName: null }); + }); + + it('per-record error gets requestId from the header', async () => { + const detokenize = resourceMethod({ + response: [{ token: 'tok1', httpCode: 404, error: 'token not found' }], + }); + const controller = new VaultController(makeFakeClient({ tokens: { detokenize } })); + const resp = await controller.detokenize(new DetokenizeRequest(['tok1'])); + expect(resp.records[0].error).toBe('token not found'); + expect(resp.records[0].requestId).toBe(REQUEST_ID); + }); + + it('returns an empty records array when the generated `response` is absent', async () => { + const detokenize = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ tokens: { detokenize } })); + const resp = await controller.detokenize(new DetokenizeRequest(['tok1'])); + expect(resp.records).toEqual([]); + }); + + it('sets requestId to null on an errored record when the raw response has no headers', async () => { + const detokenize = resourceMethod( + { response: [{ token: 'tok1', httpCode: 404, error: 'token not found' }] }, + rawResponseNoHeaders, + ); + const controller = new VaultController(makeFakeClient({ tokens: { detokenize } })); + const resp = await controller.detokenize(new DetokenizeRequest(['tok1'])); + expect(resp.records[0].error).toBe('token not found'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('rejects when the generated method throws', async () => { + const detokenize = rejectingMethod(new Error('boom-detok')); + const controller = new VaultController(makeFakeClient({ tokens: { detokenize } })); + await expect(controller.detokenize(new DetokenizeRequest(['tok1']))).rejects.toThrow('boom-detok'); + }); +}); diff --git a/flowvault/test/controller/error-records.test.ts b/flowvault/test/controller/error-records.test.ts new file mode 100644 index 00000000..8d75a0f6 --- /dev/null +++ b/flowvault/test/controller/error-records.test.ts @@ -0,0 +1,122 @@ +import { + DeleteRequest, + DeleteResponse, + GetRequest, + DeleteFilesRequest, + InsertRequest, + UpdateRequest, + DetokenizeRequest, + GetTokensRequest, + SkyflowError, +} from '../../src'; +import VaultController from '../../src/controller/vault'; +import { makeFakeClient, rejectingMethod, rawResponse, silenceConsole } from '../helpers'; + +silenceConsole(); + +describe('VaultController per-record error extraction (4xx body carrying records)', () => { + it('delete: surfaces a per-record 404 from the error body instead of throwing', async () => { + const err = { + statusCode: 404, + body: { records: [{ skyflowID: 'id1', httpCode: 404, error: 'not a valid Skyflow ID' }] }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ records: { deleteRecords: rejectingMethod(err) } })); + + const resp = await controller.delete(new DeleteRequest('cards', ['id1'])); + + expect(resp).toBeInstanceOf(DeleteResponse); + expect(resp.records[0].skyflowId).toBe('id1'); + expect(resp.records[0].httpCode).toBe(404); + expect(resp.records[0].error).toBe('not a valid Skyflow ID'); + }); + + it('get: surfaces per-record errors from the error body', async () => { + const err = { + statusCode: 404, + body: { records: [{ skyflowID: 'id1', httpCode: 404, error: 'not found' }] }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ records: { getRecords: rejectingMethod(err) } })); + + const resp = await controller.get(new GetRequest('cards', ['id1'])); + expect(resp.records[0].httpCode).toBe(404); + expect(resp.records[0].error).toBe('not found'); + }); + + it('deleteFiles: surfaces per-record errors (columns null) from the error body', async () => { + const err = { + statusCode: 404, + body: { records: [{ skyflowID: 'id1', tableName: 't', httpCode: 404, error: 'not found' }] }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ files: { deleteFiles: rejectingMethod(err) } })); + + const resp = await controller.deleteFiles( + new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'id1' }]), + ); + expect(resp.records[0].httpCode).toBe(404); + expect(resp.records[0].columns).toBeNull(); + }); + + it('insert: surfaces per-record errors from the error body', async () => { + const err = { + statusCode: 400, + body: { records: [{ tableName: '', skyflowID: null, httpCode: 400, error: 'bad record' }] }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ records: { insertRecords: rejectingMethod(err) } })); + + const resp = await controller.insert(new InsertRequest('cards', [{ data: { a: 1 } }])); + expect(resp.records[0].httpCode).toBe(400); + expect(resp.records[0].error).toBe('bad record'); + }); + + it('update: surfaces per-record errors from the error body', async () => { + const err = { + statusCode: 404, + body: { records: [{ skyflowID: null, httpCode: 404, error: 'not found' }] }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ records: { updateRecords: rejectingMethod(err) } })); + + const resp = await controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }])); + expect(resp.records[0].httpCode).toBe(404); + }); + + it('detokenize: surfaces per-record errors from the error body `response` array', async () => { + const err = { + statusCode: 404, + body: { response: [{ token: 't1', httpCode: 404, error: 'invalid token' }] }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ tokens: { detokenize: rejectingMethod(err) } })); + + const resp = await controller.detokenize(new DetokenizeRequest(['t1'])); + expect(resp.records[0].httpCode).toBe(404); + expect(resp.records[0].error).toBe('invalid token'); + }); + + it('getTokens: surfaces per-record errors from the error body', async () => { + const err = { + statusCode: 404, + body: { records: [{ value: 'v', tokenGroupName: 'g', token: '', httpCode: 404, error: 'not found' }] }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ tokens: { getTokens: rejectingMethod(err) } })); + + const resp = await controller.getTokens(new GetTokensRequest([{ value: 'v', tokenGroupName: 'g' }])); + expect(resp.records[0].httpCode).toBe(404); + }); + + it('still throws a whole-call error when the body has no records (e.g. invalid vault)', async () => { + const err = { + statusCode: 404, + body: { error: { httpCode: 404, message: 'Vault not found' } }, + rawResponse, + }; + const controller = new VaultController(makeFakeClient({ records: { deleteRecords: rejectingMethod(err) } })); + + await expect(controller.delete(new DeleteRequest('cards', ['id1']))).rejects.toBeInstanceOf(SkyflowError); + }); +}); diff --git a/flowvault/test/controller/get-tokens.test.ts b/flowvault/test/controller/get-tokens.test.ts new file mode 100644 index 00000000..f6b1b5be --- /dev/null +++ b/flowvault/test/controller/get-tokens.test.ts @@ -0,0 +1,83 @@ +import { GetTokensRequest, GetTokensResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoHeaders, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +describe('VaultController.getTokens', () => { + it('builds value/tokenGroupName records and maps the response', async () => { + const getTokens = resourceMethod({ + records: [{ value: '4111', tokenGroupName: 'cards', token: 'tok_1', httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ tokens: { getTokens } })); + + const resp = await controller.getTokens( + new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }]), + ); + + expect(resp).toBeInstanceOf(GetTokensResponse); + expect(getTokens).toHaveBeenCalledTimes(1); + const [genReq] = getTokens.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.records).toEqual([{ value: '4111', tokenGroupName: 'cards' }]); + + const record = resp.records[0]; + expect(record.value).toBe('4111'); + expect(record.tokenGroupName).toBe('cards'); + expect(record.token).toBe('tok_1'); + expect(record.httpCode).toBe(200); + expect(record.error).toBeNull(); + expect(record.requestId).toBeUndefined(); + }); + + it('defaults token to null and httpCode to 200 when absent', async () => { + const getTokens = resourceMethod({ records: [{ value: '4111', tokenGroupName: 'cards' }] }); + const controller = new VaultController(makeFakeClient({ tokens: { getTokens } })); + const resp = await controller.getTokens(new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }])); + expect(resp.records[0].token).toBeNull(); + expect(resp.records[0].httpCode).toBe(200); + }); + + it('per-record error gets requestId from the header', async () => { + const getTokens = resourceMethod({ + records: [{ value: '4111', tokenGroupName: 'cards', httpCode: 400, error: 'tokenize failed' }], + }); + const controller = new VaultController(makeFakeClient({ tokens: { getTokens } })); + const resp = await controller.getTokens(new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }])); + expect(resp.records[0].error).toBe('tokenize failed'); + expect(resp.records[0].requestId).toBe(REQUEST_ID); + }); + + it('returns an empty records array when the response has none', async () => { + const getTokens = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ tokens: { getTokens } })); + const resp = await controller.getTokens(new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }])); + expect(resp.records).toEqual([]); + }); + + it('sets requestId to null on an errored record when the raw response has no headers', async () => { + const getTokens = resourceMethod( + { records: [{ value: '4111', tokenGroupName: 'cards', httpCode: 400, error: 'tokenize failed' }] }, + rawResponseNoHeaders, + ); + const controller = new VaultController(makeFakeClient({ tokens: { getTokens } })); + const resp = await controller.getTokens(new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }])); + expect(resp.records[0].error).toBe('tokenize failed'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('rejects when the generated method throws', async () => { + const getTokens = rejectingMethod(new Error('boom-tokens')); + const controller = new VaultController(makeFakeClient({ tokens: { getTokens } })); + await expect( + controller.getTokens(new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }])), + ).rejects.toThrow('boom-tokens'); + }); +}); diff --git a/flowvault/test/controller/get.test.ts b/flowvault/test/controller/get.test.ts new file mode 100644 index 00000000..9b276fa4 --- /dev/null +++ b/flowvault/test/controller/get.test.ts @@ -0,0 +1,153 @@ +import { GetRequest, GetOptions, GetResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { makeFakeClient, resourceMethod, rejectingMethod, REQUEST_ID, silenceConsole } from '../helpers'; + +silenceConsole(); + +describe('VaultController.get', () => { + it('single-table: builds tableName/skyflowIDs and maps records', async () => { + const getRecords = resourceMethod({ + records: [ + { + skyflowID: 'id1', + tableName: 'cards', + data: { name: 'alice' }, + hashedData: { name: 'h' }, + tokens: { name: 'tok' }, + httpCode: 200, + }, + ], + }); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + + const resp = await controller.get(new GetRequest('cards', ['id1', 'id2'])); + + expect(resp).toBeInstanceOf(GetResponse); + expect(getRecords).toHaveBeenCalledTimes(1); + const [genReq] = getRecords.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.tableName).toBe('cards'); + expect(genReq.skyflowIDs).toEqual(['id1', 'id2']); + expect(genReq.records).toBeUndefined(); + + const record = resp.records[0]; + expect(record.skyflowId).toBe('id1'); + expect(record.data).toEqual({ name: 'alice' }); + expect(record.hashedData).toEqual({ name: 'h' }); + expect(record.tokens).toEqual({ name: 'tok' }); + expect(record.httpCode).toBe(200); + expect(record.error).toBeNull(); + expect(record.requestId).toBeUndefined(); + }); + + it('multi-table: builds the records[] array from request.records', async () => { + const getRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + + const req = new GetRequest('', [], [ + { tableName: 't1', ids: ['a', 'b'] }, + { tableName: 't2', ids: ['c'] }, + ]); + await controller.get(req); + + const [genReq] = getRecords.mock.calls[0]; + expect(genReq.records).toEqual([ + { tableName: 't1', skyflowIDs: ['a', 'b'] }, + { tableName: 't2', skyflowIDs: ['c'] }, + ]); + expect(genReq.tableName).toBeUndefined(); + expect(genReq.skyflowIDs).toBeUndefined(); + }); + + it('multi-table: carries per-record columns / columnRedactions / uniqueValues', async () => { + const getRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + + const req = new GetRequest('', [], [ + { + tableName: 't1', + ids: ['a'], + columns: ['name'], + columnRedactions: [{ columnName: 'ssn', redaction: 'MASKED' }], + uniqueValues: [{ email: 'x@y.com' }], + }, + { tableName: 't2', ids: ['b'] }, // no per-record fields -> only tableName/skyflowIDs + ]); + await controller.get(req); + + const [genReq] = getRecords.mock.calls[0]; + expect(genReq.records[0]).toEqual({ + tableName: 't1', + skyflowIDs: ['a'], + columns: ['name'], + columnRedactions: [{ columnName: 'ssn', redaction: 'MASKED' }], + uniqueValues: [{ data: { email: 'x@y.com' } }], + }); + expect(genReq.records[1]).toEqual({ tableName: 't2', skyflowIDs: ['b'] }); + }); + + it('maps options: columns, columnRedactions, uniqueValues->{data}, limit, offset', async () => { + const getRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + + const options = new GetOptions(); + options.setFields(['name', 'email']); + options.setColumnRedactions([{ columnName: 'ssn', redaction: 'MASKED' }]); + options.setUniqueValues([{ email: 'a@b.com' }, { email: 'c@d.com' }]); + options.setLimit(5); + options.setOffset(1); + + await controller.get(new GetRequest('cards', ['id1']), options); + + const [genReq] = getRecords.mock.calls[0]; + expect(genReq.columns).toEqual(['name', 'email']); + expect(genReq.columnRedactions).toEqual([{ columnName: 'ssn', redaction: 'MASKED' }]); + expect(genReq.uniqueValues).toEqual([{ data: { email: 'a@b.com' } }, { data: { email: 'c@d.com' } }]); + expect(genReq.limit).toBe(5); + expect(genReq.offset).toBe(1); + }); + + it('honors limit/offset of 0 (uses !== undefined guard)', async () => { + const getRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + + const options = new GetOptions(); + options.setLimit(0); + options.setOffset(0); + await controller.get(new GetRequest('cards', ['id1']), options); + + const [genReq] = getRecords.mock.calls[0]; + expect(genReq.limit).toBe(0); + expect(genReq.offset).toBe(0); + }); + + it('per-record error gets requestId from the header', async () => { + const getRecords = resourceMethod({ records: [{ httpCode: 404, error: 'not found' }] }); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + const resp = await controller.get(new GetRequest('cards', ['id1'])); + expect(resp.records[0].error).toBe('not found'); + expect(resp.records[0].requestId).toBe(REQUEST_ID); + }); + + it('returns an empty records array when the response has none', async () => { + const getRecords = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + const resp = await controller.get(new GetRequest('cards', ['id1'])); + expect(resp.records).toEqual([]); + }); + + it('sets requestId to null on an errored record when the raw response is absent', async () => { + // rawResponse null -> getRequestId short-circuits -> requestId falls back to null. + const getRecords = resourceMethod({ records: [{ httpCode: 404, error: 'not found' }] }, null); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + const resp = await controller.get(new GetRequest('cards', ['id1'])); + expect(resp.records[0].error).toBe('not found'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('rejects when the generated method throws', async () => { + const getRecords = rejectingMethod(new Error('boom-get')); + const controller = new VaultController(makeFakeClient({ records: { getRecords } })); + await expect(controller.get(new GetRequest('cards', ['id1']))).rejects.toThrow('boom-get'); + }); +}); diff --git a/flowvault/test/controller/insert.test.ts b/flowvault/test/controller/insert.test.ts new file mode 100644 index 00000000..4f004b21 --- /dev/null +++ b/flowvault/test/controller/insert.test.ts @@ -0,0 +1,171 @@ +import { InsertRequest, InsertOptions, InsertResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoHeaders, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +describe('VaultController.insert', () => { + it('maps records, calls the generated method once, and builds the request', async () => { + const insertRecords = resourceMethod({ + records: [ + { + skyflowID: 'id1', + tokens: { card_number: 'tok_1' }, + hashedData: { card_number: 'hash_1' }, + httpCode: 200, + tableName: 'cards', + }, + ], + }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + const req = new InsertRequest('cards', [{ data: { card_number: '4111' }, tokens: { card_number: 'byot_1' } }]); + const options = new InsertOptions(); + options.setUpsert({ updateType: 'UPDATE', uniqueColumns: ['card_number'] }); + + const resp = await controller.insert(req, options); + + expect(resp).toBeInstanceOf(InsertResponse); + expect(insertRecords).toHaveBeenCalledTimes(1); + + // Request built correctly (vaultID, tableName, BYOT tokens, upsert). + const [genReq, opts] = insertRecords.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.tableName).toBe('cards'); + expect(genReq.records).toEqual([{ data: { card_number: '4111' }, tokens: { card_number: 'byot_1' } }]); + expect(genReq.upsert).toEqual({ updateType: 'UPDATE', uniqueColumns: ['card_number'] }); + expect(opts.headers).toBeDefined(); + + // Response mapped field-by-field. + const record = resp.records[0]; + expect(record.skyflowId).toBe('id1'); + expect(record.tokens).toEqual({ card_number: 'tok_1' }); + expect(record.data).toBeNull(); + expect(record.hashedData).toEqual({ card_number: 'hash_1' }); + expect(record.tableName).toBe('cards'); + expect(record.httpCode).toBe(200); + expect(record.error).toBeNull(); + expect(record.requestId).toBeUndefined(); + }); + + it('omits BYOT tokens and upsert when not provided; fields fall back to data', async () => { + const insertRecords = resourceMethod({ + records: [{ skyflowID: 'id1', data: { card_number: '4111' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + const resp = await controller.insert(new InsertRequest('cards', [{ data: { card_number: '4111' } }])); + + const [genReq] = insertRecords.mock.calls[0]; + expect(genReq.records[0]).toEqual({ data: { card_number: '4111' } }); + expect(genReq.records[0].tokens).toBeUndefined(); + expect(genReq.upsert).toBeUndefined(); + expect(resp.records[0].data).toEqual({ card_number: '4111' }); // data used when no tokens + expect(resp.records[0].tokens).toBeNull(); + }); + + it('sends a per-record tableName when records carry their own table (multi-table insert)', async () => { + const insertRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + await controller.insert(new InsertRequest('cards', [ + { data: { a: 1 }, table: 'cards' }, + { data: { b: 2 }, table: 'contacts' }, + ])); + + const [genReq] = insertRecords.mock.calls[0]; + expect(genReq.tableName).toBeUndefined(); // table lives per-record, not at request level + expect(genReq.records).toEqual([ + { data: { a: 1 }, tableName: 'cards' }, + { data: { b: 2 }, tableName: 'contacts' }, + ]); + }); + + it('falls back to the request table for a record with no table of its own', async () => { + const insertRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + await controller.insert(new InsertRequest('cards', [ + { data: { a: 1 }, table: 'contacts' }, + { data: { b: 2 } }, // no per-record table -> request table 'cards' + ])); + + const [genReq] = insertRecords.mock.calls[0]; + expect(genReq.tableName).toBeUndefined(); + expect(genReq.records).toEqual([ + { data: { a: 1 }, tableName: 'contacts' }, + { data: { b: 2 }, tableName: 'cards' }, + ]); + }); + + it('sends per-record upsert when records carry it and drops the request-level upsert', async () => { + const insertRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + const options = new InsertOptions(); + options.setUpsert({ updateType: 'UPDATE', uniqueColumns: ['req'] }); // request-level — used as fallback + + await controller.insert( + new InsertRequest('cards', [ + { data: { a: 1 }, upsert: { updateType: 'UPDATE', uniqueColumns: ['email'] } }, + { data: { b: 2 } }, // no per-record upsert -> falls back to the request-level one + ]), + options, + ); + + const [genReq] = insertRecords.mock.calls[0]; + expect(genReq.upsert).toBeUndefined(); // request-level dropped once any record has its own + expect(genReq.records[0].upsert).toEqual({ updateType: 'UPDATE', uniqueColumns: ['email'] }); + expect(genReq.records[1].upsert).toEqual({ updateType: 'UPDATE', uniqueColumns: ['req'] }); + }); + + it('populates error + requestId (from header) on a per-record error', async () => { + const insertRecords = resourceMethod({ records: [{ httpCode: 400, error: 'bad record' }] }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + const resp = await controller.insert(new InsertRequest('cards', [{ data: { card_number: '4111' } }])); + + expect(resp.records[0].error).toBe('bad record'); + expect(resp.records[0].requestId).toBe(REQUEST_ID); + expect(resp.records[0].skyflowId).toBeNull(); + expect(resp.records[0].tokens).toBeNull(); + expect(resp.records[0].data).toBeNull(); + }); + + it('sets requestId to null on an errored record when the raw response has no headers', async () => { + const insertRecords = resourceMethod({ records: [{ httpCode: 400, error: 'bad' }] }, rawResponseNoHeaders); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + const resp = await controller.insert(new InsertRequest('cards', [{ data: { a: 1 } }])); + expect(resp.records[0].error).toBe('bad'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('returns an empty records array when the response has none', async () => { + const insertRecords = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + const resp = await controller.insert(new InsertRequest('cards', [{ data: { a: 1 } }])); + expect(resp.records).toEqual([]); + }); + + it('rejects (via failureResponse) when the generated method throws', async () => { + const insertRecords = rejectingMethod(new Error('network down')); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + await expect( + controller.insert(new InsertRequest('cards', [{ data: { a: 1 } }])), + ).rejects.toThrow('network down'); + }); + + it('rejects on invalid input before calling the generated method', async () => { + const insertRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + await expect(controller.insert(new InsertRequest('', [{ data: { a: 1 } }]))).rejects.toBeDefined(); + expect(insertRecords).not.toHaveBeenCalled(); + }); +}); diff --git a/flowvault/test/controller/query.test.ts b/flowvault/test/controller/query.test.ts new file mode 100644 index 00000000..e9ce3e95 --- /dev/null +++ b/flowvault/test/controller/query.test.ts @@ -0,0 +1,59 @@ +import { QueryRequest, QueryResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoId, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +describe('VaultController.query', () => { + it('maps records->fields, metadata.columns and the call-level requestId', async () => { + const executeQuery = resourceMethod({ + records: [{ data: { name: 'alice' } }, { data: { name: 'bob' } }], + metadata: { columns: ['name'] }, + }); + const controller = new VaultController(makeFakeClient({ query: { executeQuery } })); + + const resp = await controller.query(new QueryRequest('SELECT name FROM cards')); + + expect(resp).toBeInstanceOf(QueryResponse); + expect(executeQuery).toHaveBeenCalledTimes(1); + const [genReq] = executeQuery.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.query).toBe('SELECT name FROM cards'); + + expect(resp.fields).toEqual([{ name: 'alice' }, { name: 'bob' }]); + expect(resp.metadata.columns).toEqual(['name']); + expect(resp.errors).toBeNull(); + expect(resp.requestId).toBe(REQUEST_ID); + }); + + it('defaults fields to {} and columns to [] when absent; requestId falls back to ""', async () => { + const executeQuery = resourceMethod({ records: [{}, { data: null }] }, rawResponseNoId); + const controller = new VaultController(makeFakeClient({ query: { executeQuery } })); + + const resp = await controller.query(new QueryRequest('SELECT 1')); + + expect(resp.fields).toEqual([{}, {}]); + expect(resp.metadata.columns).toEqual([]); + expect(resp.requestId).toBe(''); + }); + + it('returns empty fields when the response has no records', async () => { + const executeQuery = resourceMethod({ metadata: { columns: [] } }); + const controller = new VaultController(makeFakeClient({ query: { executeQuery } })); + const resp = await controller.query(new QueryRequest('SELECT 1')); + expect(resp.fields).toEqual([]); + }); + + it('rejects when the generated method throws', async () => { + const executeQuery = rejectingMethod(new Error('boom-query')); + const controller = new VaultController(makeFakeClient({ query: { executeQuery } })); + await expect(controller.query(new QueryRequest('SELECT 1'))).rejects.toThrow('boom-query'); + }); +}); diff --git a/flowvault/test/controller/request-options.test.ts b/flowvault/test/controller/request-options.test.ts new file mode 100644 index 00000000..af09bc9c --- /dev/null +++ b/flowvault/test/controller/request-options.test.ts @@ -0,0 +1,114 @@ +import { + InsertRequest, + InsertOptions, + DeleteRequest, + DeleteOptions, + QueryRequest, + QueryOptions, + GetTokensRequest, + GetTokensOptions, + UploadFilesRequest, + UploadFilesOptions, + DeleteFilesRequest, + DeleteFilesOptions, + CustomHeaderKey, +} from '../../src'; +import VaultController from '../../src/controller/vault'; +import { makeFakeClient, resourceMethod, silenceConsole } from '../helpers'; + +silenceConsole(); + +describe('VaultController request options (timeout / retries / custom headers)', () => { + it('runs the interceptor to attach supported custom headers and includes timeout + maxRetries', async () => { + const insertRecords = resourceMethod({ records: [] }); + const controller = new VaultController( + makeFakeClient({ records: { insertRecords }, timeout: 30, maxRetries: 2 }), + ); + + const options = new InsertOptions(); + options.setInterceptor((ctx) => { + ctx.addHeader(CustomHeaderKey.REQUEST_ID_HEADER, 'req-123'); + }); + + await controller.insert(new InsertRequest('cards', [{ data: { a: 1 } }]), options); + + const [, reqOpts] = insertRecords.mock.calls[0]; + expect(reqOpts.headers['x-request-id']).toBe('req-123'); + expect(reqOpts.timeoutInSeconds).toBe(30); + expect(reqOpts.maxRetries).toBe(2); + }); + + it('passes the operation name to the interceptor context', async () => { + const insertRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + let seenOperation: string | undefined; + const options = new InsertOptions(); + options.setInterceptor((ctx) => { + seenOperation = ctx.operation; + ctx.addHeader(CustomHeaderKey.SKYFLOW_ACCOUNT_ID, 'acct-1'); + }); + + await controller.insert(new InsertRequest('cards', [{ data: { a: 1 } }]), options); + + expect(seenOperation).toBe('INSERT'); + expect(insertRecords.mock.calls[0][1].headers['x-skyflow-account-id']).toBe('acct-1'); + }); + + it('omits timeout/maxRetries and custom headers when none are configured', async () => { + const insertRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + await controller.insert(new InsertRequest('cards', [{ data: { a: 1 } }])); + + const [, reqOpts] = insertRecords.mock.calls[0]; + expect(reqOpts.timeoutInSeconds).toBeUndefined(); + expect(reqOpts.maxRetries).toBeUndefined(); + // the SDK metrics header is always present even without custom headers + expect(Object.keys(reqOpts.headers).length).toBeGreaterThan(0); + expect(reqOpts.headers['x-request-id']).toBeUndefined(); + }); + + it('wires the interceptor through delete / query / getTokens / uploadFiles / deleteFiles options', async () => { + const attach = (ctx: { addHeader: (k: CustomHeaderKey, v: string) => void }) => { + ctx.addHeader(CustomHeaderKey.REQUEST_ID_HEADER, 'r1'); + }; + + const deleteRecords = resourceMethod({ records: [] }); + const dOpts = new DeleteOptions(); + dOpts.setInterceptor(attach); + await new VaultController(makeFakeClient({ records: { deleteRecords } })) + .delete(new DeleteRequest('cards', ['id1']), dOpts); + expect(deleteRecords.mock.calls[0][1].headers['x-request-id']).toBe('r1'); + + const executeQuery = resourceMethod({ records: [], metadata: { columns: [] } }); + const qOpts = new QueryOptions(); + qOpts.setInterceptor(attach); + await new VaultController(makeFakeClient({ query: { executeQuery } })) + .query(new QueryRequest('SELECT 1'), qOpts); + expect(executeQuery.mock.calls[0][1].headers['x-request-id']).toBe('r1'); + + const getTokens = resourceMethod({ records: [] }); + const gtOpts = new GetTokensOptions(); + gtOpts.setInterceptor(attach); + await new VaultController(makeFakeClient({ tokens: { getTokens } })) + .getTokens(new GetTokensRequest([{ value: 'v', tokenGroupName: 'g' }]), gtOpts); + expect(getTokens.mock.calls[0][1].headers['x-request-id']).toBe('r1'); + + const uploadFiles = resourceMethod({ records: [] }); + const ufOpts = new UploadFilesOptions(); + ufOpts.setInterceptor(attach); + await new VaultController(makeFakeClient({ files: { uploadFiles } })) + .uploadFiles(new UploadFilesRequest([ + { tableName: 't', columns: [{ column: 'c', base64: 'aGk=', fileName: 'f.pdf' }] }, + ]), ufOpts); + expect(uploadFiles.mock.calls[0][1].headers['x-request-id']).toBe('r1'); + + const deleteFiles = resourceMethod({ records: [] }); + const dfOpts = new DeleteFilesOptions(); + dfOpts.setInterceptor(attach); + await new VaultController(makeFakeClient({ files: { deleteFiles } })) + .deleteFiles(new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'id1' }]), dfOpts); + expect(deleteFiles.mock.calls[0][1].headers['x-request-id']).toBe('r1'); + }); +}); diff --git a/flowvault/test/controller/update.test.ts b/flowvault/test/controller/update.test.ts new file mode 100644 index 00000000..f94e9bd3 --- /dev/null +++ b/flowvault/test/controller/update.test.ts @@ -0,0 +1,121 @@ +import { UpdateRequest, UpdateResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoHeaders, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +describe('VaultController.update', () => { + it('builds skyflowID/data(+BYOT tokens) records and maps the response', async () => { + const updateRecords = resourceMethod({ + records: [{ skyflowID: 'id1', tokens: { x: 'tok' }, hashedData: { x: 'h' }, tableName: 'cards', httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + + const req = new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 }, tokens: { x: 'byot' } }]); + const resp = await controller.update(req); + + expect(resp).toBeInstanceOf(UpdateResponse); + expect(updateRecords).toHaveBeenCalledTimes(1); + const [genReq] = updateRecords.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.tableName).toBe('cards'); + expect(genReq.records).toEqual([{ skyflowID: 'id1', data: { x: 1 }, tokens: { x: 'byot' } }]); + + const record = resp.records[0]; + expect(record.skyflowId).toBe('id1'); + expect(record.tokens).toEqual({ x: 'tok' }); + expect(record.data).toBeNull(); + expect(record.hashedData).toEqual({ x: 'h' }); + expect(record.tableName).toBe('cards'); + expect(record.httpCode).toBe(200); + expect(record.error).toBeNull(); + expect(record.requestId).toBeUndefined(); + }); + + it('omits BYOT tokens when not provided; fields fall back to data', async () => { + const updateRecords = resourceMethod({ records: [{ skyflowID: 'id1', data: { x: 2 }, httpCode: 200 }] }); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + + const resp = await controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 2 } }])); + + const [genReq] = updateRecords.mock.calls[0]; + expect(genReq.records[0].tokens).toBeUndefined(); + expect(resp.records[0].data).toEqual({ x: 2 }); + expect(resp.records[0].tokens).toBeNull(); + }); + + it('sends a per-record tableName when records carry their own table (falls back to request table)', async () => { + const updateRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + + await controller.update(new UpdateRequest('cards', [ + { skyflowId: 'id1', data: { x: 1 }, table: 'cards' }, + { skyflowId: 'id2', data: { y: 2 } }, // no per-record table -> request table 'cards' + ])); + + const [genReq] = updateRecords.mock.calls[0]; + expect(genReq.tableName).toBeUndefined(); + expect(genReq.records).toEqual([ + { skyflowID: 'id1', data: { x: 1 }, tableName: 'cards' }, + { skyflowID: 'id2', data: { y: 2 }, tableName: 'cards' }, + ]); + }); + + it('sends a request-level updateType when set', async () => { + const updateRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + + await controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }], 'REPLACE')); + + const [genReq] = updateRecords.mock.calls[0]; + expect(genReq.updateType).toBe('REPLACE'); + }); + + it('omits updateType when not set', async () => { + const updateRecords = resourceMethod({ records: [] }); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + + await controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }])); + + const [genReq] = updateRecords.mock.calls[0]; + expect(genReq.updateType).toBeUndefined(); + }); + + it('per-record error gets requestId from the header', async () => { + const updateRecords = resourceMethod({ records: [{ httpCode: 400, error: 'update failed' }] }); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + const resp = await controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }])); + expect(resp.records[0].error).toBe('update failed'); + expect(resp.records[0].requestId).toBe(REQUEST_ID); + }); + + it('returns an empty records array when the response has none', async () => { + const updateRecords = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + const resp = await controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }])); + expect(resp.records).toEqual([]); + }); + + it('sets requestId to null on an errored record when the raw response has no headers', async () => { + const updateRecords = resourceMethod({ records: [{ httpCode: 400, error: 'update failed' }] }, rawResponseNoHeaders); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + const resp = await controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }])); + expect(resp.records[0].error).toBe('update failed'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('rejects when the generated method throws', async () => { + const updateRecords = rejectingMethod(new Error('boom-update')); + const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); + await expect( + controller.update(new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }])), + ).rejects.toThrow('boom-update'); + }); +}); diff --git a/flowvault/test/controller/upload-files.test.ts b/flowvault/test/controller/upload-files.test.ts new file mode 100644 index 00000000..514a3199 --- /dev/null +++ b/flowvault/test/controller/upload-files.test.ts @@ -0,0 +1,316 @@ +jest.mock('fs/promises', () => ({ readFile: jest.fn() })); + +import * as fsPromises from 'fs/promises'; +import { UploadFilesRequest, UploadFilesResponse } from '../../src'; +import VaultController from '../../src/controller/vault'; +import { + makeFakeClient, + resourceMethod, + rejectingMethod, + rawResponseNoHeaders, + REQUEST_ID, + silenceConsole, +} from '../helpers'; + +silenceConsole(); + +const readFileMock = fsPromises.readFile as unknown as jest.Mock; + +/** Minimal File-like with arrayBuffer() + name, as the controller consumes. */ +function fakeFile(bytes: number[], name: string): File { + return { + name, + arrayBuffer: async () => new Uint8Array(bytes).buffer, + } as unknown as File; +} + +describe('VaultController.uploadFiles', () => { + let originalFetch: typeof global.fetch; + let fetchMock: jest.Mock; + + beforeEach(() => { + originalFetch = (global as { fetch: typeof global.fetch }).fetch; + fetchMock = jest.fn().mockResolvedValue({ ok: true, status: 200 }); + (global as unknown as { fetch: unknown }).fetch = fetchMock; + readFileMock.mockReset(); + }); + + afterEach(() => { + (global as unknown as { fetch: unknown }).fetch = originalFetch; + }); + + it('filePath source: reads bytes, infers fileName + Content-Type, PUTs with no Authorization', async () => { + readFileMock.mockResolvedValue(Buffer.from('file-bytes')); + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 'onboarding', data: { doc: 'https://signed/doc' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const req = new UploadFilesRequest([ + { tableName: 'onboarding', skyflowId: 'row1', columns: [{ column: 'doc', filePath: '/tmp/report.png' }] }, + ]); + const resp = await controller.uploadFiles(req); + + expect(resp).toBeInstanceOf(UploadFilesResponse); + expect(uploadFiles).toHaveBeenCalledTimes(1); + + // Phase A request: skyflowID + columns (fileName inferred from the path). + const [genReq] = uploadFiles.mock.calls[0]; + expect(genReq.vaultID).toBe('vault123'); + expect(genReq.records[0].skyflowID).toBe('row1'); + expect(genReq.records[0].columns).toEqual([{ column: 'doc', fileName: 'report.png' }]); + + // Phase B PUT to the signed URL. + expect(fetchMock).toHaveBeenCalledTimes(1); + const [url, init] = fetchMock.mock.calls[0]; + expect(url).toBe('https://signed/doc'); + expect(init.method).toBe('PUT'); + expect(init.headers.Authorization).toBeUndefined(); + expect(init.headers['Content-Type']).toBe('image/png'); + expect(Buffer.from(init.body).toString()).toBe('file-bytes'); + + const col = resp.records[0].columns[0]; + expect(col.column).toBe('doc'); + expect(col.fileName).toBe('report.png'); + expect(col.uploadStatus).toBe('UPLOADED'); + expect(resp.records[0].skyflowId).toBe('id1'); + expect(resp.records[0].httpCode).toBe(200); + expect(resp.records[0].error).toBeNull(); + }); + + it('base64 source: uses provided fileName, decodes bytes, infers Content-Type', async () => { + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { resume: 'https://signed/resume' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const req = new UploadFilesRequest([ + { tableName: 't', columns: [{ column: 'resume', base64: Buffer.from('hello').toString('base64'), fileName: 'resume.pdf' }] }, + ]); + const resp = await controller.uploadFiles(req); + + // No skyflowId provided -> omitted from the generated record. + const [genReq] = uploadFiles.mock.calls[0]; + expect(genReq.records[0].skyflowID).toBeUndefined(); + expect(genReq.records[0].columns).toEqual([{ column: 'resume', fileName: 'resume.pdf' }]); + + const [, init] = fetchMock.mock.calls[0]; + expect(init.headers['Content-Type']).toBe('application/pdf'); + expect(Buffer.from(init.body).toString()).toBe('hello'); + expect(resp.records[0].columns[0].uploadStatus).toBe('UPLOADED'); + }); + + it('file source: reads arrayBuffer, infers fileName from file.name', async () => { + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { pic: 'https://signed/pic' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const req = new UploadFilesRequest([ + { tableName: 't', columns: [{ column: 'pic', file: fakeFile([1, 2, 3], 'photo.jpg') }] }, + ]); + const resp = await controller.uploadFiles(req); + + const [genReq] = uploadFiles.mock.calls[0]; + expect(genReq.records[0].columns).toEqual([{ column: 'pic', fileName: 'photo.jpg' }]); + const [, init] = fetchMock.mock.calls[0]; + expect(init.headers['Content-Type']).toBe('image/jpeg'); + expect(Array.from(init.body as Uint8Array)).toEqual([1, 2, 3]); + expect(resp.records[0].columns[0].fileName).toBe('photo.jpg'); + expect(resp.records[0].columns[0].uploadStatus).toBe('UPLOADED'); + }); + + it('unknown extension falls back to application/octet-stream', async () => { + readFileMock.mockResolvedValue(Buffer.from('zipbytes')); + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { archive: 'https://signed/archive' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'archive', filePath: '/tmp/a.zip' }] }]), + ); + const [, init] = fetchMock.mock.calls[0]; + expect(init.headers['Content-Type']).toBe('application/octet-stream'); + }); + + it('FAILED when the signed-URL PUT responds not-ok', async () => { + fetchMock.mockResolvedValue({ ok: false, status: 500 }); + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { resume: 'https://signed/resume' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + + const col = resp.records[0].columns[0]; + expect(col.uploadStatus).toBe('FAILED'); + expect(col.error).toBe('Upload failed with status 500.'); + }); + + it('FAILED (with thrown message) when the signed-URL PUT throws', async () => { + fetchMock.mockRejectedValue(new Error('conn reset')); + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { resume: 'https://signed/resume' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + + const col = resp.records[0].columns[0]; + expect(col.uploadStatus).toBe('FAILED'); + expect(col.error).toBe('conn reset'); + }); + + it('SKIPPED (with record error) when the response record itself errored; requestId from header', async () => { + const uploadFiles = resourceMethod({ + records: [{ tableName: 't', httpCode: 400, error: 'row rejected' }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + + expect(fetchMock).not.toHaveBeenCalled(); + expect(resp.records[0].error).toBe('row rejected'); + expect(resp.records[0].requestId).toBe(REQUEST_ID); + const col = resp.records[0].columns[0]; + expect(col.uploadStatus).toBe('SKIPPED'); + expect(col.error).toBe('row rejected'); + }); + + it('SKIPPED (no error) when no signed URL is returned for a column', async () => { + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: {}, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + + expect(fetchMock).not.toHaveBeenCalled(); + const col = resp.records[0].columns[0]; + expect(col.uploadStatus).toBe('SKIPPED'); + expect(col.error).toBeUndefined(); + }); + + it('handles multiple columns in one record (UPLOADED + SKIPPED aligned by column name)', async () => { + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { a: 'https://signed/a' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const resp = await controller.uploadFiles( + new UploadFilesRequest([ + { + tableName: 't', + columns: [ + { column: 'a', base64: 'aGk=', fileName: 'a.pdf' }, + { column: 'b', base64: 'aGk=', fileName: 'b.pdf' }, + ], + }, + ]), + ); + + expect(fetchMock).toHaveBeenCalledTimes(1); + const cols = resp.records[0].columns; + expect(cols.find((c) => c.column === 'a')?.uploadStatus).toBe('UPLOADED'); + expect(cols.find((c) => c.column === 'b')?.uploadStatus).toBe('SKIPPED'); + }); + + it('FAILED (stringified) when the signed-URL PUT rejects with a non-Error', async () => { + fetchMock.mockRejectedValue('socket hang up'); // a thrown string, not an Error + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { resume: 'https://signed/resume' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + + const col = resp.records[0].columns[0]; + expect(col.uploadStatus).toBe('FAILED'); + expect(col.error).toBe('socket hang up'); + }); + + it('infers octet-stream for a file with no name and for a trailing-dot name', async () => { + const uploadFiles = resourceMethod({ + records: [{ skyflowID: 'id1', tableName: 't', data: { a: 'https://signed/a', b: 'https://signed/b' }, httpCode: 200 }], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + await controller.uploadFiles( + new UploadFilesRequest([ + { + tableName: 't', + columns: [ + // File-like with an undefined name and no explicit fileName -> fileName stays undefined. + { column: 'a', file: fakeFile([1], undefined as unknown as string) }, + // A trailing-dot name -> extension resolves to '' -> octet-stream. + { column: 'b', base64: 'aGk=', fileName: 'weird.' }, + ], + }, + ]), + ); + + expect(fetchMock).toHaveBeenCalledTimes(2); + for (const [, init] of fetchMock.mock.calls) { + expect(init.headers['Content-Type']).toBe('application/octet-stream'); + } + }); + + it('tolerates a response record with no matching prepared columns (prepared index absent)', async () => { + // Server returns MORE records than were sent -> prepared[1] is undefined -> []. + const uploadFiles = resourceMethod({ + records: [ + { skyflowID: 'id1', tableName: 't', data: {}, httpCode: 200 }, + { skyflowID: 'id2', tableName: 't', data: {}, httpCode: 200 }, + ], + }); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + + expect(resp.records).toHaveLength(2); + expect(resp.records[1].columns).toEqual([]); // no prepared columns for the extra record + }); + + it('returns an empty records array when the response has none', async () => { + const uploadFiles = resourceMethod({}); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + expect(resp.records).toEqual([]); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('sets requestId to null on an errored record when the raw response has no headers', async () => { + const uploadFiles = resourceMethod({ records: [{ tableName: 't', httpCode: 400, error: 'row rejected' }] }, rawResponseNoHeaders); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + const resp = await controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ); + expect(resp.records[0].error).toBe('row rejected'); + expect(resp.records[0].requestId).toBeNull(); + }); + + it('rejects when the generated uploadFiles method throws', async () => { + const uploadFiles = rejectingMethod(new Error('boom-upload')); + const controller = new VaultController(makeFakeClient({ files: { uploadFiles } })); + await expect( + controller.uploadFiles( + new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }]), + ), + ).rejects.toThrow('boom-upload'); + }); +}); diff --git a/flowvault/test/helpers.ts b/flowvault/test/helpers.ts new file mode 100644 index 00000000..144f1cb2 --- /dev/null +++ b/flowvault/test/helpers.ts @@ -0,0 +1,68 @@ +// Shared test helpers for the FlowVault hand-written layer. +// NOTE: this file is intentionally NOT a *.test.ts, so Jest does not run it as +// a suite; it only provides the mocking seam the controller/client tests reuse. +import { LogLevel, SkyflowError, SKYFLOW_ERROR_CODE } from '../src'; + +/** A rawResponse stub whose headers carry an x-request-id (matches smoke test). */ +export const REQUEST_ID = 'req-abc-123'; +export const rawResponse = { headers: new Map([['x-request-id', REQUEST_ID]]) }; + +/** A rawResponse stub carrying NO x-request-id header. */ +export const rawResponseNoId = { headers: new Map() }; + +/** A rawResponse stub with no usable headers object at all. */ +export const rawResponseNoHeaders = {}; + +/** + * Builds a fake VaultClient exposing only what the controller touches. Any + * resource client (records/tokens/files/query) is supplied via `overrides`. + */ +export function makeFakeClient(overrides: Record): any { + return { + vaultId: 'vault123', + getLogLevel: () => LogLevel.ERROR, + getToken: jest.fn().mockResolvedValue(undefined), + // Pass through real Errors; otherwise wrap. The real normalizer is covered + // by the client tests. + failureResponse: (e: unknown) => + e instanceof Error ? e : new SkyflowError(SKYFLOW_ERROR_CODE.INTERNAL_SERVER_ERROR), + ...overrides, + }; +} + +/** + * A generated-resource-method stub. Returns an object with `withRawResponse()` + * resolving to `{ data, rawResponse }`, matching the generated client contract. + */ +export function resourceMethod(data: unknown, rr: unknown = rawResponse): jest.Mock { + return jest.fn().mockReturnValue({ + withRawResponse: () => Promise.resolve({ data, rawResponse: rr }), + }); +} + +/** A generated-resource-method stub whose `withRawResponse()` rejects. */ +export function rejectingMethod(err: unknown): jest.Mock { + return jest.fn().mockReturnValue({ + withRawResponse: () => Promise.reject(err), + }); +} + +/** Build a decodable (unsigned) JWT whose `exp` is `secondsFromNow` out. */ +export function makeJwt(secondsFromNow: number): string { + const header = Buffer.from(JSON.stringify({ alg: 'none', typ: 'JWT' })).toString('base64url'); + const exp = Math.floor(Date.now() / 1000) + secondsFromNow; + const payload = Buffer.from(JSON.stringify({ exp })).toString('base64url'); + return `${header}.${payload}.sig`; +} + +/** Silence console noise from negative-path printLog() calls for a test file. */ +export function silenceConsole(): void { + beforeAll(() => { + jest.spyOn(console, 'error').mockImplementation(() => undefined); + jest.spyOn(console, 'warn').mockImplementation(() => undefined); + jest.spyOn(console, 'log').mockImplementation(() => undefined); + }); + afterAll(() => { + jest.restoreAllMocks(); + }); +} diff --git a/flowvault/test/model/options.test.ts b/flowvault/test/model/options.test.ts new file mode 100644 index 00000000..d4435a43 --- /dev/null +++ b/flowvault/test/model/options.test.ts @@ -0,0 +1,69 @@ +import { + InsertOptions, + GetOptions, + UpdateOptions, + DeleteOptions, + QueryOptions, + DetokenizeOptions, + GetTokensOptions, + UploadFilesOptions, + DeleteFilesOptions, + CustomHeaderKey, +} from '../../src'; + +describe('FlowVault options models', () => { + it('InsertOptions: upsert setter/getter + inherited interceptor', () => { + const opts = new InsertOptions(); + expect(opts.getUpsert()).toBeUndefined(); + opts.setUpsert({ updateType: 'UPDATE', uniqueColumns: ['email'] }); + expect(opts.getUpsert()).toEqual({ updateType: 'UPDATE', uniqueColumns: ['email'] }); + + expect(opts.getInterceptor()).toBeUndefined(); + const interceptor = (ctx: { addHeader: (k: CustomHeaderKey, v: string) => void }) => { + ctx.addHeader(CustomHeaderKey.REQUEST_ID_HEADER, 'v'); + }; + opts.setInterceptor(interceptor); + expect(opts.getInterceptor()).toBe(interceptor); + }); + + it('GetOptions: all setters/getters round-trip', () => { + const opts = new GetOptions(); + expect(opts.getFields()).toBeUndefined(); + expect(opts.getColumnRedactions()).toBeUndefined(); + expect(opts.getUniqueValues()).toBeUndefined(); + expect(opts.getLimit()).toBeUndefined(); + expect(opts.getOffset()).toBeUndefined(); + + opts.setFields(['name', 'email']); + opts.setColumnRedactions([{ columnName: 'ssn', redaction: 'MASKED' }]); + opts.setUniqueValues([{ email: 'a@b.com' }]); + opts.setLimit(10); + opts.setOffset(2); + + expect(opts.getFields()).toEqual(['name', 'email']); + expect(opts.getColumnRedactions()).toEqual([{ columnName: 'ssn', redaction: 'MASKED' }]); + expect(opts.getUniqueValues()).toEqual([{ email: 'a@b.com' }]); + expect(opts.getLimit()).toBe(10); + expect(opts.getOffset()).toBe(2); + }); + + it('the remaining options classes construct empty and inherit the interceptor', () => { + const classes = [ + new UpdateOptions(), + new DeleteOptions(), + new QueryOptions(), + new DetokenizeOptions(), + new GetTokensOptions(), + new UploadFilesOptions(), + new DeleteFilesOptions(), + ]; + classes.forEach((opts) => { + expect(opts.getInterceptor()).toBeUndefined(); + const interceptor = (ctx: { addHeader: (k: CustomHeaderKey, v: string) => void }) => { + ctx.addHeader(CustomHeaderKey.SKYFLOW_ACCOUNT_NAME, 'abc'); + }; + opts.setInterceptor(interceptor); + expect(opts.getInterceptor()).toBe(interceptor); + }); + }); +}); diff --git a/flowvault/test/model/requests.test.ts b/flowvault/test/model/requests.test.ts new file mode 100644 index 00000000..8c99df7c --- /dev/null +++ b/flowvault/test/model/requests.test.ts @@ -0,0 +1,128 @@ +import { + InsertRequest, + GetRequest, + UpdateRequest, + DeleteRequest, + QueryRequest, + DetokenizeRequest, + GetTokensRequest, + UploadFilesRequest, + DeleteFilesRequest, +} from '../../src'; + +describe('FlowVault request models', () => { + it('InsertRequest: ctor, getters and setters round-trip', () => { + const req = new InsertRequest('cards', [{ data: { card_number: '4111' } }]); + expect(req.tableName).toBe('cards'); + expect(req.records).toEqual([{ data: { card_number: '4111' } }]); + + req.tableName = 'people'; + req.records = [{ data: { ssn: '123' }, tokens: { ssn: 'tok' } }]; + expect(req.tableName).toBe('people'); + expect(req.records[0].tokens).toEqual({ ssn: 'tok' }); + }); + + it('GetRequest: ctor, getters and setters (table/ids/records)', () => { + const req = new GetRequest('cards', ['id1', 'id2']); + expect(req.table).toBe('cards'); + expect(req.ids).toEqual(['id1', 'id2']); + expect(req.records).toBeUndefined(); + + req.table = 'people'; + req.ids = ['id3']; + req.records = [{ tableName: 't2', ids: ['a', 'b'] }]; + expect(req.table).toBe('people'); + expect(req.ids).toEqual(['id3']); + expect(req.records).toEqual([{ tableName: 't2', ids: ['a', 'b'] }]); + + // Multi-table via ctor third arg. + const multi = new GetRequest('', [], [{ tableName: 't', ids: ['x'] }]); + expect(multi.records).toEqual([{ tableName: 't', ids: ['x'] }]); + }); + + it('UpdateRequest: ctor, getters and setters', () => { + const req = new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }]); + expect(req.tableName).toBe('cards'); + expect(req.records[0].skyflowId).toBe('id1'); + + req.tableName = 'people'; + req.records = [{ skyflowId: 'id2', data: { y: 2 }, tokens: { y: 'tok' } }]; + expect(req.tableName).toBe('people'); + expect(req.records[0].tokens).toEqual({ y: 'tok' }); + + expect(req.updateType).toBeUndefined(); + req.updateType = 'REPLACE'; + expect(req.updateType).toBe('REPLACE'); + expect(new UpdateRequest('cards', [], 'UPDATE').updateType).toBe('UPDATE'); + }); + + it('DeleteRequest: ctor, getters and setters (ids + uniqueValues)', () => { + const req = new DeleteRequest('cards', ['id1']); + expect(req.table).toBe('cards'); + expect(req.ids).toEqual(['id1']); + expect(req.uniqueValues).toBeUndefined(); + + req.table = 'people'; + req.ids = ['id2', 'id3']; + req.uniqueValues = [{ email: 'a@b.com' }]; + expect(req.table).toBe('people'); + expect(req.ids).toEqual(['id2', 'id3']); + expect(req.uniqueValues).toEqual([{ email: 'a@b.com' }]); + + const withUnique = new DeleteRequest('t', [], [{ ssn: '111' }]); + expect(withUnique.uniqueValues).toEqual([{ ssn: '111' }]); + }); + + it('QueryRequest: ctor, getter and setter', () => { + const req = new QueryRequest('SELECT * FROM cards'); + expect(req.query).toBe('SELECT * FROM cards'); + req.query = 'SELECT name FROM cards'; + expect(req.query).toBe('SELECT name FROM cards'); + }); + + it('DetokenizeRequest: ctor, getters and setters (tokens + redactions)', () => { + const req = new DetokenizeRequest(['tok1', 'tok2']); + expect(req.tokens).toEqual(['tok1', 'tok2']); + expect(req.tokenGroupRedactions).toBeUndefined(); + + req.tokens = ['tok3']; + req.tokenGroupRedactions = [{ tokenGroupName: 'g', redaction: 'MASKED' }]; + expect(req.tokens).toEqual(['tok3']); + expect(req.tokenGroupRedactions).toEqual([{ tokenGroupName: 'g', redaction: 'MASKED' }]); + + const withRedaction = new DetokenizeRequest(['t'], [{ tokenGroupName: 'cards', redaction: 'PLAIN_TEXT' }]); + expect(withRedaction.tokenGroupRedactions?.[0].tokenGroupName).toBe('cards'); + }); + + it('GetTokensRequest: ctor, getter and setter', () => { + const req = new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }]); + expect(req.records[0].value).toBe('4111'); + expect(req.records[0].tokenGroupName).toBe('cards'); + + req.records = [{ value: '5555', tokenGroupName: 'cvv' }]; + expect(req.records[0].value).toBe('5555'); + }); + + it('UploadFilesRequest: ctor, getter and setter', () => { + const req = new UploadFilesRequest([ + { tableName: 'onboarding', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }, + ]); + expect(req.records[0].tableName).toBe('onboarding'); + expect(req.records[0].columns[0].column).toBe('resume'); + + req.records = [{ tableName: 't', skyflowId: 'id1', columns: [{ column: 'c', filePath: '/p/x.png' }] }]; + expect(req.records[0].skyflowId).toBe('id1'); + expect(req.records[0].columns[0].filePath).toBe('/p/x.png'); + }); + + it('DeleteFilesRequest: ctor, getter and setter', () => { + const req = new DeleteFilesRequest([ + { tableName: 'onboarding', columns: ['resume'], skyflowId: 'id1' }, + ]); + expect(req.records[0].columns).toEqual(['resume']); + expect(req.records[0].skyflowId).toBe('id1'); + + req.records = [{ tableName: 't', columns: ['c'], uniqueValues: [{ a: 1 }] }]; + expect(req.records[0].uniqueValues).toEqual([{ a: 1 }]); + }); +}); diff --git a/flowvault/test/model/responses.test.ts b/flowvault/test/model/responses.test.ts new file mode 100644 index 00000000..1e5e6bee --- /dev/null +++ b/flowvault/test/model/responses.test.ts @@ -0,0 +1,89 @@ +import { + InsertResponse, + GetResponse, + UpdateResponse, + DeleteResponse, + QueryResponse, + DetokenizeResponse, + GetTokensResponse, + UploadFilesResponse, + DeleteFilesResponse, +} from '../../src'; + +describe('FlowVault response models', () => { + it('InsertResponse holds its records', () => { + const records = [ + { tableName: 'cards', skyflowId: 'id1', tokens: { card_number: 'tok' }, data: null, hashedData: null, httpCode: 200, error: null }, + ]; + const resp = new InsertResponse({ records }); + expect(resp.records).toBe(records); + expect(resp.records[0].skyflowId).toBe('id1'); + }); + + it('GetResponse holds its records', () => { + const records = [ + { skyflowId: 'id1', tableName: 'cards', data: { name: 'a' }, hashedData: null, tokens: null, httpCode: 200, error: null }, + ]; + const resp = new GetResponse({ records }); + expect(resp.records[0].data).toEqual({ name: 'a' }); + }); + + it('UpdateResponse holds its records', () => { + const records = [ + { skyflowId: 'id1', tableName: 'cards', tokens: { x: 1 }, data: null, hashedData: null, httpCode: 200, error: null }, + ]; + const resp = new UpdateResponse({ records }); + expect(resp.records[0].tokens).toEqual({ x: 1 }); + }); + + it('DeleteResponse holds its records (no tableName field)', () => { + const records = [{ skyflowId: 'id1', httpCode: 200, error: null }]; + const resp = new DeleteResponse({ records }); + expect(resp.records[0].skyflowId).toBe('id1'); + expect((resp.records[0] as Record).tableName).toBeUndefined(); + }); + + it('DetokenizeResponse holds its records', () => { + const records = [ + { token: 'tok1', tokenGroupName: 'cards', value: '4111', metadata: { skyflowId: 'id1', tableName: 'cards' }, httpCode: 200, error: null }, + ]; + const resp = new DetokenizeResponse({ records }); + expect(resp.records[0].value).toBe('4111'); + expect(resp.records[0].metadata).toEqual({ skyflowId: 'id1', tableName: 'cards' }); + }); + + it('GetTokensResponse holds its records', () => { + const records = [{ value: '4111', tokenGroupName: 'cards', token: 'tok', httpCode: 200, error: null }]; + const resp = new GetTokensResponse({ records }); + expect(resp.records[0].token).toBe('tok'); + }); + + it('UploadFilesResponse holds its records', () => { + const records = [ + { skyflowId: 'id1', tableName: 't', columns: [{ column: 'resume', fileName: 'r.pdf', uploadStatus: 'UPLOADED' as const }], httpCode: 200, error: null }, + ]; + const resp = new UploadFilesResponse({ records }); + expect(resp.records[0].columns[0].uploadStatus).toBe('UPLOADED'); + }); + + it('DeleteFilesResponse holds its records', () => { + const records = [ + { skyflowId: 'id1', tableName: 't', columns: [{ column: 'resume', status: 'DELETED' }], httpCode: 200, error: null }, + ]; + const resp = new DeleteFilesResponse({ records }); + expect(resp.records[0].columns?.[0].status).toBe('DELETED'); + }); + + it('QueryResponse holds fields/errors/metadata/requestId', () => { + const resp = new QueryResponse({ + fields: [{ name: 'alice' }, { name: 'bob' }], + errors: null, + metadata: { columns: ['name'] }, + requestId: 'req-1', + }); + expect(resp.fields).toEqual([{ name: 'alice' }, { name: 'bob' }]); + expect(resp.errors).toBeNull(); + expect(resp.metadata.columns).toEqual(['name']); + expect(resp.requestId).toBe('req-1'); + }); +}); diff --git a/flowvault/test/skyflow/skyflow.test.ts b/flowvault/test/skyflow/skyflow.test.ts new file mode 100644 index 00000000..4ec04bbc --- /dev/null +++ b/flowvault/test/skyflow/skyflow.test.ts @@ -0,0 +1,189 @@ +import { + Skyflow, + Env, + LogLevel, + SkyflowError, + SKYFLOW_ERROR_CODE, + getSdkIdentity, + parameterizedString, +} from '../../src'; +import VaultController from '../../src/controller/vault'; +import { silenceConsole } from '../helpers'; + +silenceConsole(); + +const vaultConfig = (vaultId: string, clusterId = 'abc-cluster') => ({ + vaultId, + clusterId, + env: Env.PROD, + credentials: { apiKey: 'sky-dummy-key' }, +}); + +function newSkyflow(vaultIds: string[] = ['vault123'], logLevel: LogLevel = LogLevel.ERROR): Skyflow { + return new Skyflow({ vaultConfigs: vaultIds.map((id) => vaultConfig(id)), logLevel }); +} + +/** Read the registry entry's VaultClient (private, accessed for assertions). */ +function clientOf(sky: Skyflow, vaultId: string): { getLogLevel: () => LogLevel; skyflowCredentials?: unknown } { + return (sky as unknown as { vaultClients: Record LogLevel; skyflowCredentials?: unknown } }> }) + .vaultClients[vaultId].client; +} + +function expectCode( + fn: () => void, + code: { httpCode: number; message: string }, + args: (string | number)[] = [], +): void { + let thrown: unknown; + try { + fn(); + } catch (e) { + thrown = e; + } + expect(thrown).toBeInstanceOf(SkyflowError); + // Parameterized messages (e.g. "%s1 is missing...") are substituted at throw time. + const expected = args.length > 0 ? parameterizedString(code.message, ...args) : code.message; + expect((thrown as SkyflowError).error?.message).toBe(expected); +} + +describe('Skyflow facade — construction and vault()', () => { + it('validates config: a config with neither vaultConfigs nor connectionConfigs throws', () => { + expectCode(() => new Skyflow({} as never), SKYFLOW_ERROR_CODE.INVALID_SKYFLOW_CONFIG); + }); + + it('throws INVALID_LOG_LEVEL for an unknown log level', () => { + expectCode( + () => new Skyflow({ vaultConfigs: [vaultConfig('vault123')], logLevel: 'BOGUS' as never }), + SKYFLOW_ERROR_CODE.INVALID_LOG_LEVEL, + ); + }); + + it('registers vaults; vault(id) and vault() (first) return a VaultController', () => { + const sky = newSkyflow(['vault123', 'vault456']); + expect(sky.vault('vault123')).toBeInstanceOf(VaultController); + expect(sky.vault('vault456')).toBeInstanceOf(VaultController); + expect(sky.vault()).toBe(sky.vault('vault123')); // no id -> first registered + }); + + it('vault() on an empty registry throws EMPTY_VAULT_CLIENTS', () => { + const sky = new Skyflow({ vaultConfigs: [], logLevel: LogLevel.ERROR }); + expectCode(() => sky.vault(), SKYFLOW_ERROR_CODE.EMPTY_VAULT_CLIENTS); + }); + + it('vault(unknownId) throws VAULT_ID_NOT_IN_CONFIG_LIST', () => { + const sky = newSkyflow(); + expectCode(() => sky.vault('nope'), SKYFLOW_ERROR_CODE.VAULT_ID_NOT_IN_CONFIG_LIST, ['nope']); + }); +}); + +describe('Skyflow facade — vault config management', () => { + it('addVaultConfig registers a new vault and rejects a duplicate id', () => { + const sky = newSkyflow(); + sky.addVaultConfig(vaultConfig('vault999')); + expect(sky.vault('vault999')).toBeInstanceOf(VaultController); + expectCode(() => sky.addVaultConfig(vaultConfig('vault999')), SKYFLOW_ERROR_CODE.VAULT_ID_EXITS_IN_CONFIG_LIST, ['vault999']); + }); + + it('addVaultConfig works with a minimal config (no env -> PROD, no credentials)', () => { + const sky = newSkyflow(); + sky.addVaultConfig({ vaultId: 'vaultMinimal', clusterId: 'min-cluster' }); + expect(sky.vault('vaultMinimal')).toBeInstanceOf(VaultController); + expect(sky.getVaultConfig('vaultMinimal').vaultId).toBe('vaultMinimal'); + }); + + it('updateVaultConfig updates an existing config and rejects an unknown id', () => { + const sky = newSkyflow(); + sky.updateVaultConfig({ vaultId: 'vault123', clusterId: 'new-cluster' }); + expect(sky.getVaultConfig('vault123').clusterId).toBe('new-cluster'); + expectCode(() => sky.updateVaultConfig({ vaultId: 'nope', clusterId: 'c' }), SKYFLOW_ERROR_CODE.VAULT_ID_NOT_IN_CONFIG_LIST, ['nope']); + }); + + it('getVaultConfig: empty id and unknown id each throw their own error', () => { + const sky = newSkyflow(); + expect(sky.getVaultConfig('vault123').vaultId).toBe('vault123'); + expectCode(() => sky.getVaultConfig(''), SKYFLOW_ERROR_CODE.EMPTY_VAULT_ID_VALIDATION); + expectCode(() => sky.getVaultConfig('nope'), SKYFLOW_ERROR_CODE.VAULT_ID_NOT_IN_CONFIG_LIST, ['nope']); + }); + + it('removeVaultConfig removes a vault; a later access and removing an unknown id throw', () => { + // Two vaults so the registry stays non-empty after one removal (otherwise + // resolveVaultEntry hits the empty-registry guard first). + const sky = newSkyflow(['vault123', 'vault456']); + sky.removeVaultConfig('vault123'); + expect(sky.vault('vault456')).toBeInstanceOf(VaultController); // the other remains + expectCode(() => sky.vault('vault123'), SKYFLOW_ERROR_CODE.VAULT_ID_NOT_IN_CONFIG_LIST, ['vault123']); + expectCode(() => sky.removeVaultConfig('vault123'), SKYFLOW_ERROR_CODE.VAULT_ID_NOT_IN_CONFIG_LIST, ['vault123']); + }); + + it('emptying the registry via removeVaultConfig makes vault() throw EMPTY_VAULT_CLIENTS', () => { + const sky = newSkyflow(); + sky.removeVaultConfig('vault123'); + expectCode(() => sky.vault('vault123'), SKYFLOW_ERROR_CODE.EMPTY_VAULT_CLIENTS); + }); +}); + +describe('Skyflow facade — log level and credentials', () => { + it('setLogLevel / updateLogLevel propagate to the vault clients; updateLogLevel returns the facade', () => { + const sky = newSkyflow(['vault123', 'vault456']); + sky.setLogLevel(LogLevel.WARN); + expect(sky.getLogLevel()).toBe(LogLevel.WARN); + expect(clientOf(sky, 'vault123').getLogLevel()).toBe(LogLevel.WARN); + + const returned = sky.updateLogLevel(LogLevel.DEBUG); + expect(returned).toBe(sky); + expect(sky.getLogLevel()).toBe(LogLevel.DEBUG); + expect(clientOf(sky, 'vault456').getLogLevel()).toBe(LogLevel.DEBUG); + }); + + it('setLogLevel rejects an invalid log level', () => { + const sky = newSkyflow(); + expectCode(() => sky.setLogLevel('NOPE' as never), SKYFLOW_ERROR_CODE.INVALID_LOG_LEVEL); + }); + + it('updateSkyflowCredentials: null throws EMPTY_CREDENTIALS; valid creds propagate and are returned', () => { + const sky = newSkyflow(); + expectCode(() => sky.updateSkyflowCredentials(null as never), SKYFLOW_ERROR_CODE.EMPTY_CREDENTIALS); + + sky.updateSkyflowCredentials({ apiKey: 'sky-new-common' } as never); + expect(sky.getSkyflowCredentials()).toEqual({ apiKey: 'sky-new-common' }); + expect(clientOf(sky, 'vault123').skyflowCredentials).toEqual({ apiKey: 'sky-new-common' }); + }); +}); + +describe('Skyflow facade — SDK identity', () => { + it('stamps the FlowVault SDK identity on import', () => { + expect(getSdkIdentity()).toEqual({ name: 'skyflow-flowvault-node', version: '1.0.0' }); + }); +}); + +describe('Skyflow facade — timeout / retry / vaultUrl config', () => { + it('passes vaultUrl override, timeout, and maxRetries to the vault client', () => { + const sky = new Skyflow({ + vaultConfigs: [{ + vaultId: 'v1', + clusterId: 'abc-cluster', + env: Env.PROD, + credentials: { apiKey: 'sky-dummy-key' }, + vaultUrl: 'https://custom.example.com', + timeout: 30, + maxRetries: 2, + }], + logLevel: LogLevel.ERROR, + }); + const client = (sky as unknown as { + vaultClients: Record; + }).vaultClients.v1.client; + expect(client.url).toBe('https://custom.example.com'); + expect(client.timeout).toBe(30); + expect(client.maxRetries).toBe(2); + }); + + it('derives the FlowVault (skyvault.*) URL from clusterId/env when vaultUrl is not set', () => { + const sky = newSkyflow(['vault123']); // vaultConfig uses clusterId 'abc-cluster', env PROD + const client = (sky as unknown as { + vaultClients: Record; + }).vaultClients.vault123.client; + // FlowVault is on skyvault.*, NOT the shared vault.* host. + expect(client.url).toBe('https://abc-cluster.skyvault.skyflowapis.com'); + }); +}); diff --git a/flowvault/test/smoke.test.ts b/flowvault/test/smoke.test.ts new file mode 100644 index 00000000..89a5e35f --- /dev/null +++ b/flowvault/test/smoke.test.ts @@ -0,0 +1,232 @@ +import { + Skyflow, + InsertRequest, + InsertOptions, + InsertResponse, + GetRequest, + GetOptions, + UpdateRequest, + DeleteRequest, + QueryRequest, + QueryResponse, + DetokenizeRequest, + DetokenizeResponse, + GetTokensRequest, + UploadFilesRequest, + UploadFilesResponse, + DeleteFilesRequest, + Env, + LogLevel, +} from '../src'; +import VaultController from '../src/controller/vault'; + +// A rawResponse stub whose headers carry an x-request-id. +const rawResponse = { headers: new Map([['x-request-id', 'req-abc-123']]) }; + +// Builds a fake VaultClient exposing only what the controller touches. +function makeFakeClient(overrides: Record) { + return { + vaultId: 'vault123', + getLogLevel: () => LogLevel.ERROR, + getToken: jest.fn().mockResolvedValue(undefined), + failureResponse: (e: unknown) => (e instanceof Error ? e : new Error(String(e))), + ...overrides, + }; +} + +function resourceMethod(data: unknown) { + return jest.fn().mockReturnValue({ + withRawResponse: () => Promise.resolve({ data, rawResponse }), + }); +} + +describe('FlowVault model classes', () => { + it('construct request classes with their fields', () => { + const insertReq = new InsertRequest('cards', [{ data: { card_number: '4111' } }]); + expect(insertReq.tableName).toBe('cards'); + expect(insertReq.records).toHaveLength(1); + + const getReq = new GetRequest('cards', ['id1', 'id2']); + expect(getReq.table).toBe('cards'); + expect(getReq.ids).toEqual(['id1', 'id2']); + + const updateReq = new UpdateRequest('cards', [{ skyflowId: 'id1', data: { x: 1 } }]); + expect(updateReq.records[0].skyflowId).toBe('id1'); + + const deleteReq = new DeleteRequest('cards', ['id1']); + expect(deleteReq.ids).toEqual(['id1']); + + const queryReq = new QueryRequest('SELECT * FROM cards'); + expect(queryReq.query).toContain('SELECT'); + + const detokenizeReq = new DetokenizeRequest(['tok1', 'tok2']); + expect(detokenizeReq.tokens).toHaveLength(2); + + const getTokensReq = new GetTokensRequest([{ value: '4111', tokenGroupName: 'cards' }]); + expect(getTokensReq.records[0].tokenGroupName).toBe('cards'); + + const uploadReq = new UploadFilesRequest([ + { tableName: 'onboarding', columns: [{ column: 'resume', base64: 'aGk=', fileName: 'r.pdf' }] }, + ]); + expect(uploadReq.records[0].tableName).toBe('onboarding'); + + const deleteFilesReq = new DeleteFilesRequest([ + { tableName: 'onboarding', columns: ['resume'], skyflowId: 'id1' }, + ]); + expect(deleteFilesReq.records[0].columns).toEqual(['resume']); + }); + + it('construct options classes via setters/getters', () => { + const insertOptions = new InsertOptions(); + insertOptions.setUpsert({ updateType: 'UPDATE', uniqueColumns: ['email'] }); + expect(insertOptions.getUpsert()).toEqual({ updateType: 'UPDATE', uniqueColumns: ['email'] }); + + const getOptions = new GetOptions(); + getOptions.setFields(['name']); + getOptions.setLimit(10); + getOptions.setOffset(2); + expect(getOptions.getFields()).toEqual(['name']); + expect(getOptions.getLimit()).toBe(10); + expect(getOptions.getOffset()).toBe(2); + }); +}); + +describe('Skyflow facade', () => { + it('builds a client from a dummy config and returns a VaultController', () => { + const client = new Skyflow({ + vaultConfigs: [ + { + vaultId: 'vault123', + clusterId: 'abc-cluster', + env: Env.PROD, + credentials: { apiKey: 'sky-dummy-key' }, + }, + ], + logLevel: LogLevel.ERROR, + }); + expect(client.vault('vault123')).toBeInstanceOf(VaultController); + // No id defaults to the first configured vault. + expect(client.vault()).toBeInstanceOf(VaultController); + expect(client.getVaultConfig('vault123').vaultId).toBe('vault123'); + }); +}); + +describe('VaultController maps generated responses', () => { + it('insert maps records to InsertResponse', async () => { + const insertRecords = resourceMethod({ + records: [{ skyflowID: 'id1', tokens: { card_number: 'tok_1' }, httpCode: 200, tableName: 'cards' }], + }); + const controller = new VaultController( + makeFakeClient({ records: { insertRecords } }) as any, + ); + + const resp = await controller.insert(new InsertRequest('cards', [{ data: { card_number: '4111' } }])); + expect(resp).toBeInstanceOf(InsertResponse); + expect(insertRecords).toHaveBeenCalledTimes(1); + expect(resp.records[0].skyflowId).toBe('id1'); + expect(resp.records[0].tokens).toEqual({ card_number: 'tok_1' }); + expect(resp.records[0].tableName).toBe('cards'); + expect(resp.records[0].httpCode).toBe(200); + expect(resp.records[0].error).toBeNull(); + expect(resp.records[0].requestId).toBeUndefined(); + }); + + it('insert attaches requestId when a record has an error', async () => { + const insertRecords = resourceMethod({ + records: [{ httpCode: 400, error: 'bad record' }], + }); + const controller = new VaultController( + makeFakeClient({ records: { insertRecords } }) as any, + ); + const resp = await controller.insert(new InsertRequest('cards', [{ data: { card_number: '4111' } }])); + expect(resp.records[0].error).toBe('bad record'); + expect(resp.records[0].requestId).toBe('req-abc-123'); + }); + + it('detokenize maps the generated `response` array to DetokenizeResponse', async () => { + const detokenize = resourceMethod({ + response: [ + { + token: 'tok1', + value: '4111', + tokenGroupName: 'cards', + httpCode: 200, + metadata: { skyflowID: 'id1', tableName: 'cards' }, + }, + ], + }); + const controller = new VaultController( + makeFakeClient({ tokens: { detokenize } }) as any, + ); + + const resp = await controller.detokenize(new DetokenizeRequest(['tok1'])); + expect(resp).toBeInstanceOf(DetokenizeResponse); + expect(resp.records[0].token).toBe('tok1'); + expect(resp.records[0].value).toBe('4111'); + expect(resp.records[0].tokenGroupName).toBe('cards'); + expect(resp.records[0].metadata).toEqual({ skyflowId: 'id1', tableName: 'cards' }); + expect(resp.records[0].httpCode).toBe(200); + }); + + it('query maps records, metadata and requestId to QueryResponse', async () => { + const executeQuery = resourceMethod({ + records: [{ data: { name: 'alice' } }, { data: { name: 'bob' } }], + metadata: { columns: ['name'] }, + }); + const controller = new VaultController( + makeFakeClient({ query: { executeQuery } }) as any, + ); + + const resp = await controller.query(new QueryRequest('SELECT name FROM cards')); + expect(resp).toBeInstanceOf(QueryResponse); + expect(resp.fields).toEqual([{ name: 'alice' }, { name: 'bob' }]); + expect(resp.metadata.columns).toEqual(['name']); + expect(resp.errors).toBeNull(); + expect(resp.requestId).toBe('req-abc-123'); + }); + + it('uploadFiles performs the two-phase upload and maps UploadFilesResponse', async () => { + const originalFetch = (global as any).fetch; + const fetchMock = jest.fn().mockResolvedValue({ ok: true, status: 200 }); + (global as any).fetch = fetchMock; + try { + const uploadFiles = resourceMethod({ + records: [ + { + skyflowID: 'id1', + tableName: 'onboarding', + data: { resume: 'https://signed.example/upload/resume' }, + httpCode: 200, + }, + ], + }); + const controller = new VaultController( + makeFakeClient({ files: { uploadFiles } }) as any, + ); + + const req = new UploadFilesRequest([ + { + tableName: 'onboarding', + columns: [{ column: 'resume', base64: Buffer.from('hello').toString('base64'), fileName: 'resume.pdf' }], + }, + ]); + const resp = await controller.uploadFiles(req); + expect(resp).toBeInstanceOf(UploadFilesResponse); + expect(uploadFiles).toHaveBeenCalledTimes(1); + // Phase B: a signed-URL PUT with no Authorization header. + expect(fetchMock).toHaveBeenCalledTimes(1); + const [url, init] = fetchMock.mock.calls[0]; + expect(url).toBe('https://signed.example/upload/resume'); + expect(init.method).toBe('PUT'); + expect(init.headers.Authorization).toBeUndefined(); + expect(init.headers['Content-Type']).toBe('application/pdf'); + + expect(resp.records[0].skyflowId).toBe('id1'); + expect(resp.records[0].columns[0].column).toBe('resume'); + expect(resp.records[0].columns[0].uploadStatus).toBe('UPLOADED'); + expect(resp.records[0].columns[0].fileName).toBe('resume.pdf'); + } finally { + (global as any).fetch = originalFetch; + } + }); +}); diff --git a/flowvault/test/validations/validations.test.ts b/flowvault/test/validations/validations.test.ts new file mode 100644 index 00000000..7b6a4430 --- /dev/null +++ b/flowvault/test/validations/validations.test.ts @@ -0,0 +1,260 @@ +import { + SkyflowError, + SKYFLOW_ERROR_CODE, + InsertRequest, + GetRequest, + GetOptions, + UpdateRequest, + DeleteRequest, + QueryRequest, + DetokenizeRequest, + GetTokensRequest, + UploadFilesRequest, + DeleteFilesRequest, +} from '../../src'; +import { + validateInsertRequest, + validateGetRequest, + validateUpdateRequest, + validateDeleteRequest, + validateQueryRequest, + validateDetokenizeRequest, + validateGetTokensRequest, + validateUploadFilesRequest, + validateDeleteFilesRequest, +} from '../../src/validations'; +import { silenceConsole } from '../helpers'; + +silenceConsole(); + +type Code = { httpCode: number; message: string }; + +/** Assert `fn` throws a SkyflowError carrying the given error code. */ +function expectCode(fn: () => void, code: Code): void { + let thrown: unknown; + try { + fn(); + } catch (e) { + thrown = e; + } + expect(thrown).toBeInstanceOf(SkyflowError); + const err = thrown as SkyflowError; + expect(err.error?.message).toBe(code.message); + expect(err.error?.httpCode).toBe(code.httpCode); +} + +describe('validateInsertRequest', () => { + it('throws on invalid request / empty table / empty records / empty data', () => { + expectCode(() => validateInsertRequest({} as unknown as InsertRequest), SKYFLOW_ERROR_CODE.INVALID_INSERT_REQUEST); + expectCode(() => validateInsertRequest(null as unknown as InsertRequest), SKYFLOW_ERROR_CODE.INVALID_INSERT_REQUEST); + expectCode(() => validateInsertRequest(new InsertRequest('', [{ data: { a: 1 } }])), SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME); + expectCode(() => validateInsertRequest(new InsertRequest('t', [])), SKYFLOW_ERROR_CODE.EMPTY_RECORD_IN_INSERT); + expectCode( + () => validateInsertRequest(new InsertRequest('t', [{ data: null as unknown as Record }])), + SKYFLOW_ERROR_CODE.EMPTY_DATA_IN_INSERT, + ); + }); + + it('does not throw on valid input', () => { + expect(() => validateInsertRequest(new InsertRequest('t', [{ data: { a: 1 } }]))).not.toThrow(); + }); +}); + +describe('validateGetRequest', () => { + it('throws on invalid request / empty table / empty ids (single-table)', () => { + expectCode(() => validateGetRequest({} as unknown as GetRequest), SKYFLOW_ERROR_CODE.INVALID_GET_REQUEST); + expectCode(() => validateGetRequest(new GetRequest('', ['id'])), SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME); + expectCode(() => validateGetRequest(new GetRequest('t', [])), SKYFLOW_ERROR_CODE.EMPTY_IDS_IN_GET); + }); + + it('does not throw for valid single-table or multi-table input', () => { + expect(() => validateGetRequest(new GetRequest('t', ['id']))).not.toThrow(); + expect(() => validateGetRequest(new GetRequest('', [], [{ tableName: 't', ids: ['i'] }]))).not.toThrow(); + }); + + it('allows empty ids when uniqueValues are provided on GetOptions', () => { + const options = new GetOptions(); + options.setUniqueValues([{ email: 'jane@example.com' }]); + expect(() => validateGetRequest(new GetRequest('t', []), options)).not.toThrow(); + }); +}); + +describe('validateUpdateRequest', () => { + it('throws on invalid request / empty table / empty records / missing skyflowId / empty data', () => { + expectCode(() => validateUpdateRequest({} as unknown as UpdateRequest), SKYFLOW_ERROR_CODE.INVALID_UPDATE_REQUEST); + expectCode( + () => validateUpdateRequest(new UpdateRequest('', [{ skyflowId: 'i', data: { a: 1 } }])), + SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME, + ); + expectCode(() => validateUpdateRequest(new UpdateRequest('t', [])), SKYFLOW_ERROR_CODE.INVALID_RECORD_IN_UPDATE); + expectCode( + () => validateUpdateRequest(new UpdateRequest('t', [{ skyflowId: '', data: { a: 1 } }])), + SKYFLOW_ERROR_CODE.MISSING_SKYFLOW_ID_IN_UPDATE, + ); + expectCode( + () => validateUpdateRequest(new UpdateRequest('t', [{ skyflowId: 'i', data: null as unknown as Record }])), + SKYFLOW_ERROR_CODE.EMPTY_DATA_IN_UPDATE, + ); + }); + + it('does not throw on valid input', () => { + expect(() => validateUpdateRequest(new UpdateRequest('t', [{ skyflowId: 'i', data: { a: 1 } }]))).not.toThrow(); + }); + + it('treats a null record entry as a missing skyflowId', () => { + expectCode( + () => validateUpdateRequest(new UpdateRequest('t', [null as unknown as { skyflowId: string; data: Record }])), + SKYFLOW_ERROR_CODE.MISSING_SKYFLOW_ID_IN_UPDATE, + ); + }); +}); + +describe('validateDeleteRequest', () => { + it('throws on invalid request / empty table / no ids nor uniqueValues', () => { + expectCode(() => validateDeleteRequest({} as unknown as DeleteRequest), SKYFLOW_ERROR_CODE.INVALID_DELETE_REQUEST); + expectCode(() => validateDeleteRequest(new DeleteRequest('', ['id'])), SKYFLOW_ERROR_CODE.EMPTY_TABLE_NAME); + expectCode(() => validateDeleteRequest(new DeleteRequest('t', [])), SKYFLOW_ERROR_CODE.EMPTY_DELETE_IDS); + }); + + it('does not throw with ids or with uniqueValues', () => { + expect(() => validateDeleteRequest(new DeleteRequest('t', ['id']))).not.toThrow(); + expect(() => validateDeleteRequest(new DeleteRequest('t', [], [{ email: 'a@b.com' }]))).not.toThrow(); + }); +}); + +describe('validateQueryRequest', () => { + it('throws on invalid request / empty query', () => { + expectCode(() => validateQueryRequest({} as unknown as QueryRequest), SKYFLOW_ERROR_CODE.INVALID_QUERY_REQUEST); + expectCode(() => validateQueryRequest(new QueryRequest('')), SKYFLOW_ERROR_CODE.EMPTY_QUERY); + }); + + it('does not throw on valid input', () => { + expect(() => validateQueryRequest(new QueryRequest('SELECT 1'))).not.toThrow(); + }); +}); + +describe('validateDetokenizeRequest', () => { + it('throws on invalid request / empty tokens / empty token entry', () => { + expectCode(() => validateDetokenizeRequest({} as unknown as DetokenizeRequest), SKYFLOW_ERROR_CODE.INVALID_DETOKENIZE_REQUEST); + expectCode(() => validateDetokenizeRequest(new DetokenizeRequest([])), SKYFLOW_ERROR_CODE.EMPTY_TOKENS_IN_DETOKENIZE); + expectCode(() => validateDetokenizeRequest(new DetokenizeRequest([''])), SKYFLOW_ERROR_CODE.EMPTY_TOKEN_IN_DETOKENIZE); + }); + + it('does not throw on valid input', () => { + expect(() => validateDetokenizeRequest(new DetokenizeRequest(['tok']))).not.toThrow(); + }); +}); + +describe('validateGetTokensRequest', () => { + it('throws on invalid request / empty records / missing tokenGroupName', () => { + expectCode(() => validateGetTokensRequest({} as unknown as GetTokensRequest), SKYFLOW_ERROR_CODE.INVALID_TOKENIZE_REQUEST); + expectCode(() => validateGetTokensRequest(new GetTokensRequest([])), SKYFLOW_ERROR_CODE.EMPTY_VALUES_IN_TOKENIZE); + expectCode( + () => validateGetTokensRequest(new GetTokensRequest([{ value: 'v', tokenGroupName: '' }])), + SKYFLOW_ERROR_CODE.EMPTY_COLUMN_GROUP_IN_TOKENIZE, + ); + }); + + it('does not throw on valid input', () => { + expect(() => validateGetTokensRequest(new GetTokensRequest([{ value: 'v', tokenGroupName: 'g' }]))).not.toThrow(); + }); + + it('treats a null record entry as a missing tokenGroupName', () => { + expectCode( + () => validateGetTokensRequest(new GetTokensRequest([null as unknown as { value: unknown; tokenGroupName: string }])), + SKYFLOW_ERROR_CODE.EMPTY_COLUMN_GROUP_IN_TOKENIZE, + ); + }); +}); + +describe('validateUploadFilesRequest', () => { + it('throws on invalid request / empty records', () => { + expectCode(() => validateUploadFilesRequest({} as unknown as UploadFilesRequest), SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST); + expectCode(() => validateUploadFilesRequest(new UploadFilesRequest([])), SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST); + }); + + it('throws on missing table / columns / column name', () => { + expectCode( + () => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: '', columns: [{ column: 'c', base64: 'x', fileName: 'f.pdf' }] }])), + SKYFLOW_ERROR_CODE.MISSING_TABLE_IN_UPLOAD_FILE, + ); + expectCode( + () => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [] }])), + SKYFLOW_ERROR_CODE.MISSING_COLUMN_NAME_IN_UPLOAD_FILE, + ); + expectCode( + () => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [{ column: '', base64: 'x', fileName: 'f.pdf' }] }])), + SKYFLOW_ERROR_CODE.MISSING_COLUMN_NAME_IN_UPLOAD_FILE, + ); + }); + + it('throws unless exactly one of filePath/base64/file is provided', () => { + expectCode( + () => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'c' }] }])), + SKYFLOW_ERROR_CODE.MISSING_FILE_SOURCE_IN_UPLOAD_FILE, + ); + expectCode( + () => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'c', base64: 'x', filePath: '/p', fileName: 'f.pdf' }] }])), + SKYFLOW_ERROR_CODE.MISSING_FILE_SOURCE_IN_UPLOAD_FILE, + ); + }); + + it('throws when base64 is provided without a fileName', () => { + expectCode( + () => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'c', base64: 'x' }] }])), + SKYFLOW_ERROR_CODE.MISSING_FILE_NAME_FOR_BASE64, + ); + }); + + it('treats a null record entry as a missing table', () => { + expectCode( + () => validateUploadFilesRequest(new UploadFilesRequest([null as unknown as { tableName: string; columns: [] }])), + SKYFLOW_ERROR_CODE.MISSING_TABLE_IN_UPLOAD_FILE, + ); + }); + + it('does not throw for valid base64 / filePath / file sources', () => { + const fakeFile = { name: 'f.pdf', arrayBuffer: async () => new ArrayBuffer(0) } as unknown as File; + expect(() => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'c', base64: 'x', fileName: 'f.pdf' }] }]))).not.toThrow(); + expect(() => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'c', filePath: '/p/x.png' }] }]))).not.toThrow(); + expect(() => validateUploadFilesRequest(new UploadFilesRequest([{ tableName: 't', columns: [{ column: 'c', file: fakeFile }] }]))).not.toThrow(); + }); +}); + +describe('validateDeleteFilesRequest', () => { + it('throws on invalid request / empty records / missing table / missing columns', () => { + expectCode(() => validateDeleteFilesRequest({} as unknown as DeleteFilesRequest), SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST); + expectCode(() => validateDeleteFilesRequest(new DeleteFilesRequest([])), SKYFLOW_ERROR_CODE.INVALID_FILE_UPLOAD_REQUEST); + expectCode( + () => validateDeleteFilesRequest(new DeleteFilesRequest([{ tableName: '', columns: ['c'], skyflowId: 'i' }])), + SKYFLOW_ERROR_CODE.MISSING_TABLE_IN_UPLOAD_FILE, + ); + expectCode( + () => validateDeleteFilesRequest(new DeleteFilesRequest([{ tableName: 't', columns: [], skyflowId: 'i' }])), + SKYFLOW_ERROR_CODE.MISSING_COLUMN_NAME_IN_UPLOAD_FILE, + ); + }); + + it('requires exactly one of skyflowId / uniqueValues', () => { + expectCode( + () => validateDeleteFilesRequest(new DeleteFilesRequest([{ tableName: 't', columns: ['c'] }])), + SKYFLOW_ERROR_CODE.MISSING_SKYFLOW_ID_IN_UPLOAD_FILE, + ); + expectCode( + () => validateDeleteFilesRequest(new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'i', uniqueValues: [{ a: 1 }] }])), + SKYFLOW_ERROR_CODE.MISSING_SKYFLOW_ID_IN_UPLOAD_FILE, + ); + }); + + it('treats a null record entry as a missing table', () => { + expectCode( + () => validateDeleteFilesRequest(new DeleteFilesRequest([null as unknown as { tableName: string; columns: string[] }])), + SKYFLOW_ERROR_CODE.MISSING_TABLE_IN_UPLOAD_FILE, + ); + }); + + it('does not throw with skyflowId only or uniqueValues only', () => { + expect(() => validateDeleteFilesRequest(new DeleteFilesRequest([{ tableName: 't', columns: ['c'], skyflowId: 'i' }]))).not.toThrow(); + expect(() => validateDeleteFilesRequest(new DeleteFilesRequest([{ tableName: 't', columns: ['c'], uniqueValues: [{ a: 1 }] }]))).not.toThrow(); + }); +}); diff --git a/flowvault/tsconfig.dts.json b/flowvault/tsconfig.dts.json new file mode 100644 index 00000000..1588e812 --- /dev/null +++ b/flowvault/tsconfig.dts.json @@ -0,0 +1,32 @@ +{ + "compilerOptions": { + "target": "es2019", + "module": "commonjs", + "moduleResolution": "node", + "lib": ["es2015", "es2017", "es2019", "dom"], + "esModuleInterop": true, + "allowSyntheticDefaultImports": true, + "allowJs": true, + "strict": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": false, + "noImplicitAny": false, + "declaration": true, + "emitDeclarationOnly": true, + "composite": false, + "incremental": false, + "stripInternal": true, + "types": ["node"], + "rootDir": "." + }, + "include": ["src/**/*.ts", "_generated_/**/*.ts"], + "exclude": [ + "node_modules", + "lib", + "dist", + "test", + "_generated_/rest/src/Client.ts", + "_generated_/rest/src/index.ts" + ] +} diff --git a/flowvault/tsconfig.json b/flowvault/tsconfig.json new file mode 100644 index 00000000..02e52479 --- /dev/null +++ b/flowvault/tsconfig.json @@ -0,0 +1,36 @@ +{ + "compilerOptions": { + "composite": true, + "sourceMap": true, + "declaration": true, + "declarationMap": true, + "stripInternal": true, + "noImplicitAny": false, + "target": "es6", + "lib": ["es2015", "es2017", "es2019", "dom"], + "esModuleInterop": true, + "module": "commonjs", + "moduleResolution": "node", + "removeComments": false, + "allowSyntheticDefaultImports": true, + "allowJs": true, + "strict": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": false, + "isolatedModules": false, + "noEmit": false, + "rootDir": ".", + "outDir": "lib", + "types": ["node"] + }, + "references": [{ "path": "../common" }], + "include": ["src/**/*.ts", "_generated_/**/*.ts"], + "exclude": [ + "node_modules", + "lib", + "test", + "_generated_/rest/src/Client.ts", + "_generated_/rest/src/index.ts" + ] +} diff --git a/flowvault/tsconfig.test.json b/flowvault/tsconfig.test.json new file mode 100644 index 00000000..8db19576 --- /dev/null +++ b/flowvault/tsconfig.test.json @@ -0,0 +1,19 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "composite": false, + "declaration": false, + "declarationMap": false, + "incremental": false, + "noEmit": true, + "isolatedModules": true, + "rootDir": "." + }, + "include": ["src/**/*.ts", "test/**/*.ts", "_generated_/**/*.ts"], + "exclude": [ + "node_modules", + "lib", + "_generated_/rest/src/Client.ts", + "_generated_/rest/src/index.ts" + ] +} diff --git a/flowvault/tsup.config.ts b/flowvault/tsup.config.ts new file mode 100644 index 00000000..067ef014 --- /dev/null +++ b/flowvault/tsup.config.ts @@ -0,0 +1,19 @@ +import { defineConfig } from 'tsup'; + +// Bundles the published entry, inlining the private `skyflow-common` workspace +// package so the tarball is self-contained (no skyflow-common dependency ships). +// Real npm dependencies stay external (declared in package.json "dependencies"). +export default defineConfig({ + entry: { index: 'src/index.ts' }, + format: ['cjs'], + outDir: 'dist', + target: 'es2019', + platform: 'node', + noExternal: ['skyflow-common'], + tsconfig: 'tsconfig.dts.json', + dts: true, + sourcemap: true, + clean: true, + splitting: false, + shims: false, +}); diff --git a/package-lock.json b/package-lock.json index 5a83cb00..659f4d7a 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,26 +1,18 @@ { - "name": "skyflow-node", - "version": "2.1.1-dev.09ed991", + "name": "skyflow-node-sdk", + "version": "0.0.0", "lockfileVersion": 3, "requires": true, "packages": { "": { - "name": "skyflow-node", - "version": "2.1.1-dev.09ed991", + "name": "skyflow-node-sdk", + "version": "0.0.0", "license": "MIT", - "dependencies": { - "dotenv": "16.6.1", - "form-data": "4.0.6", - "form-data-encoder": "4.1.0", - "formdata-node": "6.0.3", - "js-base64": "3.7.7", - "jsonwebtoken": "9.0.3", - "jwt-decode": "4.0.0", - "node-fetch": "2.7.0", - "qs": "6.15.2", - "readable-stream": "4.7.0", - "url-join": "4.0.1" - }, + "workspaces": [ + "common", + "skyvault", + "flowvault" + ], "devDependencies": { "@babel/plugin-proposal-decorators": "7.29.0", "@babel/plugin-transform-object-assign": "7.27.1", @@ -41,10 +33,45 @@ "jest": "29.7.0", "prettier": "3.6.2", "ts-jest": "29.4.11", + "tsup": "^8.5.1", "typescript": "5.7.3", "typescript-eslint": "8.59.3" } }, + "common": { + "name": "skyflow-common", + "version": "0.0.0", + "license": "MIT", + "dependencies": { + "dotenv": "16.6.1", + "js-base64": "3.7.7", + "jsonwebtoken": "9.0.3", + "jwt-decode": "4.0.0", + "node-fetch": "2.7.0", + "qs": "6.15.2", + "readable-stream": "4.7.0", + "url-join": "4.0.1" + } + }, + "flowvault": { + "name": "skyflow-flowvault-node", + "version": "1.0.0", + "license": "MIT", + "dependencies": { + "dotenv": "16.6.1", + "form-data": "4.0.6", + "form-data-encoder": "4.1.0", + "formdata-node": "6.0.3", + "js-base64": "3.7.7", + "node-fetch": "2.7.0", + "qs": "6.15.2", + "readable-stream": "4.7.0", + "url-join": "4.0.1" + }, + "devDependencies": { + "skyflow-common": "*" + } + }, "node_modules/@babel/code-frame": { "version": "7.29.7", "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.7.tgz", @@ -2571,6 +2598,448 @@ "node": ">=20" } }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz", + "integrity": "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.27.7.tgz", + "integrity": "sha512-jbPXvB4Yj2yBV7HUfE2KHe4GJX51QplCN1pGbYjvsyCZbQmies29EoJbkEc+vYuU5o45AfQn37vZlyXy4YJ8RQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.27.7.tgz", + "integrity": "sha512-62dPZHpIXzvChfvfLJow3q5dDtiNMkwiRzPylSCfriLvZeq0a1bWChrGx/BbUbPwOrsWKMn8idSllklzBy+dgQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.27.7.tgz", + "integrity": "sha512-x5VpMODneVDb70PYV2VQOmIUUiBtY3D3mPBG8NxVk5CogneYhkR7MmM3yR/uMdITLrC1ml/NV1rj4bMJuy9MCg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.27.7.tgz", + "integrity": "sha512-5lckdqeuBPlKUwvoCXIgI2D9/ABmPq3Rdp7IfL70393YgaASt7tbju3Ac+ePVi3KDH6N2RqePfHnXkaDtY9fkw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.27.7.tgz", + "integrity": "sha512-rYnXrKcXuT7Z+WL5K980jVFdvVKhCHhUwid+dDYQpH+qu+TefcomiMAJpIiC2EM3Rjtq0sO3StMV/+3w3MyyqQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.27.7.tgz", + "integrity": "sha512-B48PqeCsEgOtzME2GbNM2roU29AMTuOIN91dsMO30t+Ydis3z/3Ngoj5hhnsOSSwNzS+6JppqWsuhTp6E82l2w==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.27.7.tgz", + "integrity": "sha512-jOBDK5XEjA4m5IJK3bpAQF9/Lelu/Z9ZcdhTRLf4cajlB+8VEhFFRjWgfy3M1O4rO2GQ/b2dLwCUGpiF/eATNQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.27.7.tgz", + "integrity": "sha512-RkT/YXYBTSULo3+af8Ib0ykH8u2MBh57o7q/DAs3lTJlyVQkgQvlrPTnjIzzRPQyavxtPtfg0EopvDyIt0j1rA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.27.7.tgz", + "integrity": "sha512-RZPHBoxXuNnPQO9rvjh5jdkRmVizktkT7TCDkDmQ0W2SwHInKCAV95GRuvdSvA7w4VMwfCjUiPwDi0ZO6Nfe9A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.27.7.tgz", + "integrity": "sha512-GA48aKNkyQDbd3KtkplYWT102C5sn/EZTY4XROkxONgruHPU72l+gW+FfF8tf2cFjeHaRbWpOYa/uRBz/Xq1Pg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.27.7.tgz", + "integrity": "sha512-a4POruNM2oWsD4WKvBSEKGIiWQF8fZOAsycHOt6JBpZ+JN2n2JH9WAv56SOyu9X5IqAjqSIPTaJkqN8F7XOQ5Q==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.27.7.tgz", + "integrity": "sha512-KabT5I6StirGfIz0FMgl1I+R1H73Gp0ofL9A3nG3i/cYFJzKHhouBV5VWK1CSgKvVaG4q1RNpCTR2LuTVB3fIw==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.27.7.tgz", + "integrity": "sha512-gRsL4x6wsGHGRqhtI+ifpN/vpOFTQtnbsupUF5R5YTAg+y/lKelYR1hXbnBdzDjGbMYjVJLJTd2OFmMewAgwlQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.27.7.tgz", + "integrity": "sha512-hL25LbxO1QOngGzu2U5xeXtxXcW+/GvMN3ejANqXkxZ/opySAZMrc+9LY/WyjAan41unrR3YrmtTsUpwT66InQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.27.7.tgz", + "integrity": "sha512-2k8go8Ycu1Kb46vEelhu1vqEP+UeRVj2zY1pSuPdgvbd5ykAw82Lrro28vXUrRmzEsUV0NzCf54yARIK8r0fdw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.27.7.tgz", + "integrity": "sha512-hzznmADPt+OmsYzw1EE33ccA+HPdIqiCRq7cQeL1Jlq2gb1+OyWBkMCrYGBJ+sxVzve2ZJEVeePbLM2iEIZSxA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.27.7.tgz", + "integrity": "sha512-b6pqtrQdigZBwZxAn1UpazEisvwaIDvdbMbmrly7cDTMFnw/+3lVxxCTGOrkPVnsYIosJJXAsILG9XcQS+Yu6w==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.27.7.tgz", + "integrity": "sha512-OfatkLojr6U+WN5EDYuoQhtM+1xco+/6FSzJJnuWiUw5eVcicbyK3dq5EeV/QHT1uy6GoDhGbFpprUiHUYggrw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.27.7.tgz", + "integrity": "sha512-AFuojMQTxAz75Fo8idVcqoQWEHIXFRbOc1TrVcFSgCZtQfSdc1RXgB3tjOn/krRHENUB4j00bfGjyl2mJrU37A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.27.7.tgz", + "integrity": "sha512-+A1NJmfM8WNDv5CLVQYJ5PshuRm/4cI6WMZRg1by1GwPIQPCTs1GLEUHwiiQGT5zDdyLiRM/l1G0Pv54gvtKIg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openharmony-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.27.7.tgz", + "integrity": "sha512-+KrvYb/C8zA9CU/g0sR6w2RBw7IGc5J2BPnc3dYc5VJxHCSF1yNMxTV5LQ7GuKteQXZtspjFbiuW5/dOj7H4Yw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.27.7.tgz", + "integrity": "sha512-ikktIhFBzQNt/QDyOL580ti9+5mL/YZeUPKU2ivGtGjdTYoqz6jObj6nOMfhASpS4GU4Q/Clh1QtxWAvcYKamA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.27.7.tgz", + "integrity": "sha512-7yRhbHvPqSpRUV7Q20VuDwbjW5kIMwTHpptuUzV+AA46kiPze5Z7qgt6CLCK3pWFrHeNfDd1VKgyP4O+ng17CA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.27.7.tgz", + "integrity": "sha512-SmwKXe6VHIyZYbBLJrhOoCJRB/Z1tckzmgTLfFYOfpMAx63BJEaL9ExI8x7v0oAO3Zh6D/Oi1gVxEYr5oUCFhw==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.27.7.tgz", + "integrity": "sha512-56hiAJPhwQ1R4i+21FVF7V8kSD5zZTdHcVuRFMW0hn753vVfQN8xlx4uOPT4xoGH0Z/oVATuR82AiqSTDIpaHg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, "node_modules/@eslint-community/eslint-utils": { "version": "4.9.1", "resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.9.1.tgz", @@ -3633,6 +4102,373 @@ "resolve": "~1.22.2" } }, + "node_modules/@napi-rs/lzma-linux-x64-gnu": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/@napi-rs/lzma-linux-x64-gnu/-/lzma-linux-x64-gnu-1.5.1.tgz", + "integrity": "sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^22.20 || ^24.12 || >=25" + } + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.64.6.tgz", + "integrity": "sha512-FkO19Fowv/OQeJZPVfdetJaI3/VpbZmBsnmAlR/rQ+b3QwfiGQiOpVegpSRH331kelvHTVw9hXYG3DwjAaRzyQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.64.6.tgz", + "integrity": "sha512-si5/Etm6MphPc2xYHnk455Shls/hoMjTy3mhXge4OPmvCw7TwC4KP4fKokAlKLV4XaQvLvgSHfafPcblLMYYdQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.64.6.tgz", + "integrity": "sha512-PQCl3q7b0BgvVt72i2xrWM6I5KliwwVe4GpWW5QDGyUQawDS9aPZss/aGyFqBmxdRxlE8wDc9CaxiQ5DHRSp/Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.64.6.tgz", + "integrity": "sha512-l9DFl1Itb1YYdSrP1K3U8ih44AOHa/p+Q4WMMDfGvH5qVWp+ajfgS1DlYo4yCRh8wJ2ZJv0Fqccf+ULAmMFjmg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.64.6.tgz", + "integrity": "sha512-bR4GR3205N9+od0+l5Dd9/iZTDpJkIe6d6Ydp2qbcQZJLmKBdYQQRp8gbWpbMt92twKJY66o1vK4NkoKQODIrA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.64.6.tgz", + "integrity": "sha512-XTVNaDzcFl2TeBglvxdGgsCv9JGtAGRAoJENmGBR/XvDH2mA2K0IXc2aMfP0dR8mexSXSKifJ85FixoXX1wfTw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.64.6.tgz", + "integrity": "sha512-qWc6qbc+25ugG9vqPR9P422KJBbVfZ05mjcMggoq8tsfWP+eRfC+/tiDdIlRF5VL2pDVZWN9zAhWsmu7R1+7Dw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.64.6.tgz", + "integrity": "sha512-V4Nwkp8844X/Xnww3PI1LYS0YPei8XQEVG70R+5MPZGO7AhYoun4hCN7Svs4U3wbQLZbwiz+9xRFhUZZdRl8Ww==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.64.6.tgz", + "integrity": "sha512-xb72A1aPaaIsp+a88VtRae/ovIEXislbUMyp7b9AQQU7ZkP2m/+ik6DW9TQcJyw5R8/RIailF9zExYwkjR++UA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.64.6.tgz", + "integrity": "sha512-pbcs1BVBwC43hZkjhjlF8Rx8m5G0KnhLZs9Azy/gqccP6UsNSzJr0hKGIAVPqzAH1O/NdPxxC9GVGb4msQg/aw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-gnu": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.64.6.tgz", + "integrity": "sha512-EKOC8os6Vsg487ZLKUqzLF99Jsmnq6tu7O7mAELakNpcA+8KadktFf/JYdRuHe/DzCUXmN38pUKu4XBcKlqGbQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-musl": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.64.6.tgz", + "integrity": "sha512-v/SQ68GnSlr/HyjB79sNXesGDwkOPu9JjbHfmWaymCiM+nZsQ9TstG1KfA4MuEvTr2d3PoY8PhFxj9bt0e32JA==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-gnu": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.64.6.tgz", + "integrity": "sha512-G3pmliMQ7r15zFFNwBlaT/xTdSLi4lfgI7CUrQueyKYheNVhrd8JrWCE6Ll8xf4AA1RZFqq+mNJYntiElsVOyA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-musl": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.64.6.tgz", + "integrity": "sha512-XQTB/IPoB/wG6x/wu9uomAt8YH61IAr5iZTubMiad/lu88sotq2ocz9hJd556+Jz2kIkF7ZQZIsUuWE3OWqtcA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.64.6.tgz", + "integrity": "sha512-O5mpokth6+ydMuwiGtY8fIjvOBfH7u6SzmDNmZ3k6GK5C6iEAC2q2ZN/T5L9jYILx0IcCT0KnfMDihQ5yQFSBw==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.64.6.tgz", + "integrity": "sha512-aIFxbAD9v3TvIaEDsKmukjcDg1JS4/tFXw8SHyQQgzOwzWCQ4cyDK7Ju1pQKqUXDFkEu/BZCFDCDATsKhQcB/Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.64.6.tgz", + "integrity": "sha512-fuNo1UwBc6BxnHDOxxguQL/InhheMN3RJcJMb0b4xP8r5Q3CNCrJbVg0eCIFuJ4e/zAfjR2gYbOfbzO/G7CIyw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.64.6.tgz", + "integrity": "sha512-05v6wFKDKJPZs9xlytVrrRu6IbRuoSvlH9xNwEOafXsKyjBFZPvAcZz2VIAB/M+lblYeKa9iOYC4RDVdfcAAaw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.64.6.tgz", + "integrity": "sha512-wJXKcdMjvkc/FaavZXNP1yW4i0pfOmhZAIWRSLG2yJ0PB0TWi7dkLErhX0u0vptDcXKKO/m1Y/YTeuyvb2MXwg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-openbsd-x64": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.64.6.tgz", + "integrity": "sha512-cwg7/h0ThvJwy8CBkmACdiHAxRWpbsCmIsjOt+Zzx0hTnIcSUmKYEV03bGg+hD827OnwUsbKh1ZZDW754xWyBA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ] + }, + "node_modules/@rollup/rollup-openharmony-arm64": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.64.6.tgz", + "integrity": "sha512-T0RkYIJbdYXJEQFw+tMBO09KDFdgmNxkHsof3wWAABrvnx15Yh8RytmrgYy9+q9cBeBNZF66MMQZC+5nzEz6vg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.64.6.tgz", + "integrity": "sha512-KlfpmF1AAIdAWxt8QQNClbpLFlAUfYOzqIuJ7IXEStMHpFRuH78RG6FjqPb3Rqhh0K89F5npvG+/i3Fjl6IveQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.64.6.tgz", + "integrity": "sha512-MrTLjd7Jlt9/ZavJZYG61KQv9Iy9bzbY+yXy73Xvwoa2uP66wvDShkyBr5Chz/yrE6lbD/zHMuTmKQVNQqBApg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-gnu": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.64.6.tgz", + "integrity": "sha512-8fWDH1CkPI8GdBKAxzHDza7UmgficsKo2+MVbvUgOGhB0629w5sHMg3Qu5y7TZsrYKHHWpz8fla4KvoT7WsO3Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.64.6.tgz", + "integrity": "sha512-9/c9PWdr5GCIHkkxiHM1HjNBQMVdftP4O8QTcoUAaRza2FSqJ4axbf8QAnBuE6BIGQDl71SaRPeDdDYid5ecTw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, "node_modules/@rushstack/node-core-library": { "version": "5.23.1", "resolved": "https://registry.npmjs.org/@rushstack/node-core-library/-/node-core-library-5.23.1.tgz", @@ -4341,6 +5177,13 @@ "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, + "node_modules/any-promise": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/any-promise/-/any-promise-1.3.0.tgz", + "integrity": "sha512-7UvmKalWRt1wgjL1RrGxoSJW/0QZFIegpeGvZG9kjp8vrRu55XTHbwnqq2GpXm9uLbcuhxm3IqX9OB4MZR1b2A==", + "dev": true, + "license": "MIT" + }, "node_modules/anymatch": { "version": "3.1.3", "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz", @@ -4758,6 +5601,32 @@ "dev": true, "license": "MIT" }, + "node_modules/bundle-require": { + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/bundle-require/-/bundle-require-5.1.0.tgz", + "integrity": "sha512-3WrrOuZiyaaZPWiEt4G3+IffISVC9HYlWueJEBWED4ZH4aIAC2PnkdnuRrR94M+w6yGWn4AglWtJtBI8YqvgoA==", + "dev": true, + "license": "MIT", + "dependencies": { + "load-tsconfig": "^0.2.3" + }, + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" + }, + "peerDependencies": { + "esbuild": ">=0.18" + } + }, + "node_modules/cac": { + "version": "6.7.14", + "resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz", + "integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, "node_modules/call-bind-apply-helpers": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", @@ -4867,6 +5736,22 @@ "node": ">=10" } }, + "node_modules/chokidar": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz", + "integrity": "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==", + "dev": true, + "license": "MIT", + "dependencies": { + "readdirp": "^4.0.1" + }, + "engines": { + "node": ">= 14.16.0" + }, + "funding": { + "url": "https://paulmillr.com/funding/" + } + }, "node_modules/ci-info": { "version": "3.9.0", "resolved": "https://registry.npmjs.org/ci-info/-/ci-info-3.9.0.tgz", @@ -5003,6 +5888,23 @@ "dev": true, "license": "MIT" }, + "node_modules/confbox": { + "version": "0.1.8", + "resolved": "https://registry.npmjs.org/confbox/-/confbox-0.1.8.tgz", + "integrity": "sha512-RMtmw0iFkeR4YV+fUOSucriAQNb9g8zFR52MWCtl+cCZOFRNL6zeB395vPzFhEjjn4fMxXudmELnl/KF/WrK6w==", + "dev": true, + "license": "MIT" + }, + "node_modules/consola": { + "version": "3.4.2", + "resolved": "https://registry.npmjs.org/consola/-/consola-3.4.2.tgz", + "integrity": "sha512-5IKcdX0nnYavi6G7TtOhwkYzyjfJlatbjMjuLSfE2kYT5pMDOilZ4OvMhi637CcDICTmz3wARPoyhqyX1Y+XvA==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^14.18.0 || >=16.10.0" + } + }, "node_modules/convert-source-map": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", @@ -5527,6 +6429,48 @@ "node": ">= 0.4" } }, + "node_modules/esbuild": { + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz", + "integrity": "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.27.7", + "@esbuild/android-arm": "0.27.7", + "@esbuild/android-arm64": "0.27.7", + "@esbuild/android-x64": "0.27.7", + "@esbuild/darwin-arm64": "0.27.7", + "@esbuild/darwin-x64": "0.27.7", + "@esbuild/freebsd-arm64": "0.27.7", + "@esbuild/freebsd-x64": "0.27.7", + "@esbuild/linux-arm": "0.27.7", + "@esbuild/linux-arm64": "0.27.7", + "@esbuild/linux-ia32": "0.27.7", + "@esbuild/linux-loong64": "0.27.7", + "@esbuild/linux-mips64el": "0.27.7", + "@esbuild/linux-ppc64": "0.27.7", + "@esbuild/linux-riscv64": "0.27.7", + "@esbuild/linux-s390x": "0.27.7", + "@esbuild/linux-x64": "0.27.7", + "@esbuild/netbsd-arm64": "0.27.7", + "@esbuild/netbsd-x64": "0.27.7", + "@esbuild/openbsd-arm64": "0.27.7", + "@esbuild/openbsd-x64": "0.27.7", + "@esbuild/openharmony-arm64": "0.27.7", + "@esbuild/sunos-x64": "0.27.7", + "@esbuild/win32-arm64": "0.27.7", + "@esbuild/win32-ia32": "0.27.7", + "@esbuild/win32-x64": "0.27.7" + } + }, "node_modules/escalade": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", @@ -6006,6 +6950,18 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/fix-dts-default-cjs-exports": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/fix-dts-default-cjs-exports/-/fix-dts-default-cjs-exports-1.0.1.tgz", + "integrity": "sha512-pVIECanWFC61Hzl2+oOCtoJ3F17kglZC/6N94eRWycFgBH35hHx0Li604ZIzhseh97mf2p0cv7vVrOZGoqhlEg==", + "dev": true, + "license": "MIT", + "dependencies": { + "magic-string": "^0.30.17", + "mlly": "^1.7.4", + "rollup": "^4.34.8" + } + }, "node_modules/flat-cache": { "version": "4.0.1", "resolved": "https://registry.npmjs.org/flat-cache/-/flat-cache-4.0.1.tgz", @@ -7986,6 +8942,16 @@ "dev": true, "license": "MIT" }, + "node_modules/joycon": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/joycon/-/joycon-3.1.1.tgz", + "integrity": "sha512-34wB/Y7MW7bzjKRjUKTa46I2Z7eV62Rkhva+KkopW7Qvv/OSWBqvkSY7vusOPrNuZcUG3tApvdVgNB8POj3SPw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=10" + } + }, "node_modules/js-base64": { "version": "3.7.7", "resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.7.7.tgz", @@ -8204,6 +9170,19 @@ "node": ">= 0.8.0" } }, + "node_modules/lilconfig": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/lilconfig/-/lilconfig-3.1.3.tgz", + "integrity": "sha512-/vlFKAoH5Cgt3Ie+JLhRbwOsCQePABiU3tJ1egGvyQ+33R/vcwM2Zl2QR/LzjsBeItPt3oSVXapn+m4nQDvpzw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14" + }, + "funding": { + "url": "https://github.com/sponsors/antonk52" + } + }, "node_modules/lines-and-columns": { "version": "1.2.4", "resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz", @@ -8211,6 +9190,16 @@ "dev": true, "license": "MIT" }, + "node_modules/load-tsconfig": { + "version": "0.2.5", + "resolved": "https://registry.npmjs.org/load-tsconfig/-/load-tsconfig-0.2.5.tgz", + "integrity": "sha512-IXO6OCs9yg8tMKzfPZ1YmheJbZCiEsnBdcB03l0OcfK9prKnJb96siuHCr5Fl37/yo9DnKU+TLpxzTUspw9shg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" + } + }, "node_modules/locate-path": { "version": "6.0.0", "resolved": "https://registry.npmjs.org/locate-path/-/locate-path-6.0.0.tgz", @@ -8300,6 +9289,16 @@ "yallist": "^3.0.2" } }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, "node_modules/make-dir": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz", @@ -8446,12 +9445,37 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/mlly": { + "version": "1.8.2", + "resolved": "https://registry.npmjs.org/mlly/-/mlly-1.8.2.tgz", + "integrity": "sha512-d+ObxMQFmbt10sretNDytwt85VrbkhhUA/JBGm1MPaWJ65Cl4wOgLaB1NYvJSZ0Ef03MMEU/0xpPMXUIQ29UfA==", + "dev": true, + "license": "MIT", + "dependencies": { + "acorn": "^8.16.0", + "pathe": "^2.0.3", + "pkg-types": "^1.3.1", + "ufo": "^1.6.3" + } + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "license": "MIT" }, + "node_modules/mz": { + "version": "2.7.0", + "resolved": "https://registry.npmjs.org/mz/-/mz-2.7.0.tgz", + "integrity": "sha512-z81GNO7nnYMEhrGh9LeymoE4+Yr0Wn5McHIZMK5cfQCl+NDX08sCZgUc9/6MHni9IWuFLm1Z3HTCXu2z9fN62Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "any-promise": "^1.0.0", + "object-assign": "^4.0.1", + "thenify-all": "^1.0.0" + } + }, "node_modules/natural-compare": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/natural-compare/-/natural-compare-1.4.0.tgz", @@ -8545,6 +9569,16 @@ "node": ">=8" } }, + "node_modules/object-assign": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", + "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/object-inspect": { "version": "1.13.4", "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", @@ -8712,6 +9746,13 @@ "dev": true, "license": "MIT" }, + "node_modules/pathe": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", + "dev": true, + "license": "MIT" + }, "node_modules/picocolors": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", @@ -8811,6 +9852,61 @@ "node": ">=8" } }, + "node_modules/pkg-types": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/pkg-types/-/pkg-types-1.3.1.tgz", + "integrity": "sha512-/Jm5M4RvtBFVkKWRu2BLUTNP8/M2a+UwuAX+ae4770q1qVGtfjG+WTCupoZixokjmHiry8uI+dlY8KXYV5HVVQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "confbox": "^0.1.8", + "mlly": "^1.7.4", + "pathe": "^2.0.1" + } + }, + "node_modules/postcss-load-config": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/postcss-load-config/-/postcss-load-config-6.0.1.tgz", + "integrity": "sha512-oPtTM4oerL+UXmx+93ytZVN82RrlY/wPUV8IeDxFrzIjXOLF1pN+EmKPLbubvKHT2HC20xXsCAH2Z+CKV6Oz/g==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "lilconfig": "^3.1.1" + }, + "engines": { + "node": ">= 18" + }, + "peerDependencies": { + "jiti": ">=1.21.0", + "postcss": ">=8.0.9", + "tsx": "^4.8.1", + "yaml": "^2.4.2" + }, + "peerDependenciesMeta": { + "jiti": { + "optional": true + }, + "postcss": { + "optional": true + }, + "tsx": { + "optional": true + }, + "yaml": { + "optional": true + } + } + }, "node_modules/prelude-ls": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/prelude-ls/-/prelude-ls-1.2.1.tgz", @@ -8940,6 +10036,20 @@ "node": "^12.22.0 || ^14.17.0 || >=16.0.0" } }, + "node_modules/readdirp": { + "version": "4.1.2", + "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-4.1.2.tgz", + "integrity": "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 14.18.0" + }, + "funding": { + "type": "individual", + "url": "https://paulmillr.com/funding/" + } + }, "node_modules/regenerate": { "version": "1.4.2", "resolved": "https://registry.npmjs.org/regenerate/-/regenerate-1.4.2.tgz", @@ -9073,6 +10183,52 @@ "node": ">=10" } }, + "node_modules/rollup": { + "version": "4.64.6", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.64.6.tgz", + "integrity": "sha512-fX1Vy+UNKdXn+xKDnu/HBXbHVr1AY8cSdQj9BU4kLv5aFCMqGpvrVbyUMmjN3DdzWhSAuTbcX8PCNEVaumthYA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.9" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@napi-rs/lzma-linux-x64-gnu": "1.5.1", + "@rollup/rollup-android-arm-eabi": "4.64.6", + "@rollup/rollup-android-arm64": "4.64.6", + "@rollup/rollup-darwin-arm64": "4.64.6", + "@rollup/rollup-darwin-x64": "4.64.6", + "@rollup/rollup-freebsd-arm64": "4.64.6", + "@rollup/rollup-freebsd-x64": "4.64.6", + "@rollup/rollup-linux-arm-gnueabihf": "4.64.6", + "@rollup/rollup-linux-arm-musleabihf": "4.64.6", + "@rollup/rollup-linux-arm64-gnu": "4.64.6", + "@rollup/rollup-linux-arm64-musl": "4.64.6", + "@rollup/rollup-linux-loong64-gnu": "4.64.6", + "@rollup/rollup-linux-loong64-musl": "4.64.6", + "@rollup/rollup-linux-ppc64-gnu": "4.64.6", + "@rollup/rollup-linux-ppc64-musl": "4.64.6", + "@rollup/rollup-linux-riscv64-gnu": "4.64.6", + "@rollup/rollup-linux-riscv64-musl": "4.64.6", + "@rollup/rollup-linux-s390x-gnu": "4.64.6", + "@rollup/rollup-linux-x64-gnu": "4.64.6", + "@rollup/rollup-linux-x64-musl": "4.64.6", + "@rollup/rollup-openbsd-x64": "4.64.6", + "@rollup/rollup-openharmony-arm64": "4.64.6", + "@rollup/rollup-win32-arm64-msvc": "4.64.6", + "@rollup/rollup-win32-ia32-msvc": "4.64.6", + "@rollup/rollup-win32-x64-gnu": "4.64.6", + "@rollup/rollup-win32-x64-msvc": "4.64.6", + "fsevents": "~2.3.2" + } + }, "node_modules/safe-buffer": { "version": "5.2.1", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", @@ -9212,6 +10368,18 @@ "dev": true, "license": "MIT" }, + "node_modules/skyflow-common": { + "resolved": "common", + "link": true + }, + "node_modules/skyflow-flowvault-node": { + "resolved": "flowvault", + "link": true + }, + "node_modules/skyflow-node": { + "resolved": "skyvault", + "link": true + }, "node_modules/slash": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/slash/-/slash-3.0.0.tgz", @@ -9390,6 +10558,39 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/sucrase": { + "version": "3.35.1", + "resolved": "https://registry.npmjs.org/sucrase/-/sucrase-3.35.1.tgz", + "integrity": "sha512-DhuTmvZWux4H1UOnWMB3sk0sbaCVOoQZjv8u1rDoTV0HTdGem9hkAZtl4JZy8P2z4Bg0nT+YMeOFyVr4zcG5Tw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/gen-mapping": "^0.3.2", + "commander": "^4.0.0", + "lines-and-columns": "^1.1.6", + "mz": "^2.7.0", + "pirates": "^4.0.1", + "tinyglobby": "^0.2.11", + "ts-interface-checker": "^0.1.9" + }, + "bin": { + "sucrase": "bin/sucrase", + "sucrase-node": "bin/sucrase-node" + }, + "engines": { + "node": ">=16 || 14 >=14.17" + } + }, + "node_modules/sucrase/node_modules/commander": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/commander/-/commander-4.1.1.tgz", + "integrity": "sha512-NOKm8xhkzAjzFx8B2v5OAHT+u5pRQc2UCa2Vq9jYL/31o2wi9mxBA7LIFs3sV5VSC49z6pEhfbMULvShKj26WA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 6" + } + }, "node_modules/supports-color": { "version": "8.1.1", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-8.1.1.tgz", @@ -9465,6 +10666,36 @@ "node": "*" } }, + "node_modules/thenify": { + "version": "3.3.1", + "resolved": "https://registry.npmjs.org/thenify/-/thenify-3.3.1.tgz", + "integrity": "sha512-RVZSIV5IG10Hk3enotrhvz0T9em6cyHBLkH/YAZuKqd8hRkKhSfCGIcP2KUY0EPxndzANBmNllzWPwak+bheSw==", + "dev": true, + "license": "MIT", + "dependencies": { + "any-promise": "^1.0.0" + } + }, + "node_modules/thenify-all": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/thenify-all/-/thenify-all-1.6.0.tgz", + "integrity": "sha512-RNxQH/qI8/t3thXJDwcstUO4zeqo64+Uy/+sNVRBx4Xn2OX+OZ9oP+iJnNFqplFra2ZUVeKCSa2oVWi3T4uVmA==", + "dev": true, + "license": "MIT", + "dependencies": { + "thenify": ">= 3.1.0 < 4" + }, + "engines": { + "node": ">=0.8" + } + }, + "node_modules/tinyexec": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz", + "integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==", + "dev": true, + "license": "MIT" + }, "node_modules/tinyglobby": { "version": "0.2.16", "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", @@ -9502,6 +10733,16 @@ "node": ">=8.0" } }, + "node_modules/tree-kill": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/tree-kill/-/tree-kill-1.2.2.tgz", + "integrity": "sha512-L0Orpi8qGpRG//Nd+H90vFB+3iHnue1zSSGmNOOCh1GLJ7rUKVwV2HvijphGQS2UmhUZewS9VgvxYIdgr+fG1A==", + "dev": true, + "license": "MIT", + "bin": { + "tree-kill": "cli.js" + } + }, "node_modules/ts-api-utils": { "version": "2.5.0", "resolved": "https://registry.npmjs.org/ts-api-utils/-/ts-api-utils-2.5.0.tgz", @@ -9515,6 +10756,13 @@ "typescript": ">=4.8.4" } }, + "node_modules/ts-interface-checker": { + "version": "0.1.13", + "resolved": "https://registry.npmjs.org/ts-interface-checker/-/ts-interface-checker-0.1.13.tgz", + "integrity": "sha512-Y/arvbn+rrz3JCKl9C4kVNfTfSm2/mEp5FSz5EsZSANGPSlQrpRI5M4PKF+mJnE52jOO90PnPSc3Ur3bTQw0gA==", + "dev": true, + "license": "Apache-2.0" + }, "node_modules/ts-jest": { "version": "29.4.11", "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.4.11.tgz", @@ -9594,6 +10842,69 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/tsup": { + "version": "8.5.1", + "resolved": "https://registry.npmjs.org/tsup/-/tsup-8.5.1.tgz", + "integrity": "sha512-xtgkqwdhpKWr3tKPmCkvYmS9xnQK3m3XgxZHwSUjvfTjp7YfXe5tT3GgWi0F2N+ZSMsOeWeZFh7ZZFg5iPhing==", + "dev": true, + "license": "MIT", + "dependencies": { + "bundle-require": "^5.1.0", + "cac": "^6.7.14", + "chokidar": "^4.0.3", + "consola": "^3.4.0", + "debug": "^4.4.0", + "esbuild": "^0.27.0", + "fix-dts-default-cjs-exports": "^1.0.0", + "joycon": "^3.1.1", + "picocolors": "^1.1.1", + "postcss-load-config": "^6.0.1", + "resolve-from": "^5.0.0", + "rollup": "^4.34.8", + "source-map": "^0.7.6", + "sucrase": "^3.35.0", + "tinyexec": "^0.3.2", + "tinyglobby": "^0.2.11", + "tree-kill": "^1.2.2" + }, + "bin": { + "tsup": "dist/cli-default.js", + "tsup-node": "dist/cli-node.js" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "@microsoft/api-extractor": "^7.36.0", + "@swc/core": "^1", + "postcss": "^8.4.12", + "typescript": ">=4.5.0" + }, + "peerDependenciesMeta": { + "@microsoft/api-extractor": { + "optional": true + }, + "@swc/core": { + "optional": true + }, + "postcss": { + "optional": true + }, + "typescript": { + "optional": true + } + } + }, + "node_modules/tsup/node_modules/source-map": { + "version": "0.7.6", + "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.7.6.tgz", + "integrity": "sha512-i5uvt8C3ikiWeNZSVZNWcfZPItFQOsYTUAOkcUPGd8DqDy1uOUikjt5dG+uRlwyvR108Fb9DOd4GvXfT0N2/uQ==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">= 12" + } + }, "node_modules/type-check": { "version": "0.4.0", "resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz", @@ -9668,6 +10979,13 @@ "typescript": ">=4.8.4 <6.1.0" } }, + "node_modules/ufo": { + "version": "1.6.4", + "resolved": "https://registry.npmjs.org/ufo/-/ufo-1.6.4.tgz", + "integrity": "sha512-JFNbkD1Svwe0KvGi8GOeLcP4kAWQ609twvCdcHxq1oSL8svv39ZuSvajcD8B+5D0eL4+s1Is2D/O6KN3qcTeRA==", + "dev": true, + "license": "MIT" + }, "node_modules/uglify-js": { "version": "3.19.3", "resolved": "https://registry.npmjs.org/uglify-js/-/uglify-js-3.19.3.tgz", @@ -10004,6 +11322,27 @@ "funding": { "url": "https://github.com/sponsors/sindresorhus" } + }, + "skyvault": { + "name": "skyflow-node", + "version": "2.1.2", + "license": "MIT", + "dependencies": { + "dotenv": "16.6.1", + "form-data": "4.0.6", + "form-data-encoder": "4.1.0", + "formdata-node": "6.0.3", + "js-base64": "3.7.7", + "jsonwebtoken": "9.0.3", + "jwt-decode": "4.0.0", + "node-fetch": "2.7.0", + "qs": "6.15.2", + "readable-stream": "4.7.0", + "url-join": "4.0.1" + }, + "devDependencies": { + "skyflow-common": "*" + } } } } diff --git a/package.json b/package.json index 662f66e4..be6621ae 100644 --- a/package.json +++ b/package.json @@ -1,20 +1,25 @@ { - "name": "skyflow-node", - "version": "2.1.2", - "description": "Skyflow SDK for Node.js", - "main": "./lib/index.js", - "module": "./lib/index.js", + "name": "skyflow-node-sdk", + "version": "0.0.0", + "private": true, + "description": "Skyflow Node SDK monorepo (common / skyvault / flowvault)", + "workspaces": [ + "common", + "skyvault", + "flowvault" + ], "scripts": { - "test": "jest --coverage", - "contract-snapshot": "npm run build && npx api-extractor run --local", - "contract-snapshot-verify": "npm run build && npx api-extractor run", - "build": "tsc", + "build": "tsc -b", + "build:clean": "tsc -b --clean", + "build:dist": "npm run build:dist --workspace skyvault && npm run build:dist --workspace flowvault", + "contract-snapshot": "npm run build && npm run build:dist && npm run contract-snapshot --workspace skyvault && npm run contract-snapshot --workspace flowvault", + "contract-snapshot-verify": "npm run build && npm run build:dist && npm run contract-snapshot-verify --workspace skyvault && npm run contract-snapshot-verify --workspace flowvault", + "test": "npm run test --workspaces --if-present", "eslint": "eslint '**/*.{js,ts}' --max-warnings 0", "prettier": "prettier --list-different '**/*.{js,ts}'", "lint": "npm run eslint", "lint-fix": "prettier --write '**/*.{js,ts}' && eslint --fix '**/*.{js,ts}'", "spellcheck": "cspell '**/*.{ts,js,md}'", - "docs-gen": "typedoc && node scripts/docs-script/markdown-gen.js && npx ts-node scripts/docs-script/processMarkdown.ts", "patch-generated-secrets": "node scripts/patch-generated-secrets.js", "prepare": "git rev-parse --is-inside-work-tree >/dev/null 2>&1 && git config core.hooksPath .githooks || true" }, @@ -22,9 +27,6 @@ "type": "git", "url": "https://github.com/skyflowapi/skyflow-node-sdk.git" }, - "browser": { - "child_process": false - }, "author": "", "license": "MIT", "keywords": [ @@ -36,19 +38,6 @@ "PII", "tokenization" ], - "dependencies": { - "dotenv": "16.6.1", - "form-data": "4.0.6", - "form-data-encoder": "4.1.0", - "formdata-node": "6.0.3", - "js-base64": "3.7.7", - "jsonwebtoken": "9.0.3", - "jwt-decode": "4.0.0", - "node-fetch": "2.7.0", - "qs": "6.15.2", - "readable-stream": "4.7.0", - "url-join": "4.0.1" - }, "devDependencies": { "@babel/plugin-proposal-decorators": "7.29.0", "@babel/plugin-transform-object-assign": "7.27.1", @@ -69,6 +58,7 @@ "jest": "29.7.0", "prettier": "3.6.2", "ts-jest": "29.4.11", + "tsup": "^8.5.1", "typescript": "5.7.3", "typescript-eslint": "8.59.3" } diff --git a/scripts/bump_version.sh b/scripts/bump_version.sh index 9fe7e66b..fcfc0b13 100755 --- a/scripts/bump_version.sh +++ b/scripts/bump_version.sh @@ -1,19 +1,63 @@ -Version=$1 -SEMVER=$Version +#!/usr/bin/env bash +# +# Bump the version of one published SDK in the monorepo. +# +# Usage: +# bump_version.sh [sha] +# +# skyflow-node -> bumps skyvault/package.json +# skyflow-flowvault-node -> bumps flowvault/package.json +# semver, e.g. 2.1.3 or 1.0.0 +# [sha] optional short commit SHA; when present the version becomes +# "-dev." (internal/JFrog pre-release builds) +# +set -euo pipefail -if [ -z $2 ] -then - echo "Bumping package version to $1" +PRODUCT=$1 +SEMVER=$2 +SHA=${3:-} - sed -E "s/\"version\": .+/\"version\": \"$SEMVER\",/g" package.json > tempfile && cat tempfile > package.json && rm -f tempfile - echo -------------------------- - echo "Done, Package now at $1" +case "$PRODUCT" in + skyflow-node) + PKG_DIR="skyvault" + ;; + skyflow-flowvault-node) + PKG_DIR="flowvault" + ;; + *) + echo "Unknown product '$PRODUCT'. Expected 'skyflow-node' or 'skyflow-flowvault-node'." >&2 + exit 1 + ;; +esac +PKG_JSON="$PKG_DIR/package.json" + +if [ ! -f "$PKG_JSON" ]; then + echo "Cannot find $PKG_JSON" >&2 + exit 1 +fi + +if [ -z "$SHA" ]; then + TARGET="$SEMVER" else - echo "Bumping package version to $1-dev.$2" + TARGET="$SEMVER-dev.$SHA" +fi + +echo "Bumping $PRODUCT ($PKG_JSON) to $TARGET" + +# Replace only the first "version" line (the package's own version field). +sed -E "0,/\"version\": .+/s//\"version\": \"$TARGET\",/" "$PKG_JSON" > "$PKG_JSON.tmp" \ + && cat "$PKG_JSON.tmp" > "$PKG_JSON" \ + && rm -f "$PKG_JSON.tmp" - sed -E "s/\"version\": .+/\"version\": \"$SEMVER-dev.$2\",/g" package.json > tempfile && cat tempfile > package.json && rm -f tempfile +# Keep the in-source SDK version constant in sync (self-reported identity). +VERSION_TS="$PKG_DIR/src/version.ts" +if [ -f "$VERSION_TS" ]; then + sed -E "s/export const SDK_VERSION = \".+\";/export const SDK_VERSION = \"$TARGET\";/" "$VERSION_TS" > "$VERSION_TS.tmp" \ + && cat "$VERSION_TS.tmp" > "$VERSION_TS" \ + && rm -f "$VERSION_TS.tmp" + echo "Updated $VERSION_TS to $TARGET" +fi - echo -------------------------- - echo "Done, Package now at $1-dev.$2" -fi \ No newline at end of file +echo -------------------------- +echo "Done, $PRODUCT now at $TARGET" diff --git a/skyvault/.babelrc b/skyvault/.babelrc new file mode 100644 index 00000000..eb19681f --- /dev/null +++ b/skyvault/.babelrc @@ -0,0 +1,14 @@ +{ + "presets": [ + ["@babel/preset-env", { + "targets": { + "browsers": ["last 2 versions", "ie >= 10"] + } + }] + ], + "sourceType": "unambiguous", + "plugins": [ + "@babel/plugin-transform-object-assign", + "@babel/plugin-transform-runtime" + ] +} \ No newline at end of file diff --git a/skyvault/LICENSE b/skyvault/LICENSE new file mode 100644 index 00000000..189f34b5 --- /dev/null +++ b/skyvault/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2021 Skyflow Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/skyvault/README.md b/skyvault/README.md new file mode 100644 index 00000000..5f765097 --- /dev/null +++ b/skyvault/README.md @@ -0,0 +1,1325 @@ +# Skyflow Node.js SDK + +> **This is the current, recommended version of the Skyflow SDK.** V2.1.0 brings flexible auth, multi-vault support, builder patterns, native data types, and rich error diagnostics. +> +> Migrating from v1? See the **[Migration Guide](docs/migrate_to_v2.md)** for step-by-step instructions. V1 is in maintenance mode and will reach End of Life on October 31, 2026. + +Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow SDK for Node.js, Deno, Bun, and Cloudflare Workers. + +[![CI](https://img.shields.io/static/v1?label=CI&message=passing&color=green?style=plastic&logo=github)](https://github.com/skyflowapi/skyflow-node/actions) +[![GitHub release](https://badge.fury.io/js/skyflow-node.svg)](https://www.npmjs.com/package/skyflow-node) +[![License](https://img.shields.io/github/license/skyflowapi/skyflow-node)](https://github.com/skyflowapi/skyflow-node/blob/main/LICENSE) + +## Table of contents + +- [Skyflow Node.js SDK](#skyflow-nodejs-sdk) + - [Table of contents](#table-of-contents) + - [Overview](#overview) + - [Installation](#installation) + - [Require](#require) + - [ES modules](#es-modules) + - [All imports](#all-imports) + - [Quickstart](#quickstart) + - [Before you begin](#before-you-begin) + - [Authenticate](#authenticate) + - [API Key](#api-key) + - [Bearer Token (static)](#bearer-token-static) + - [Initialize the client](#initialize-the-client) + - [Insert data into the vault, get tokens back](#insert-data-into-the-vault-get-tokens-back) + - [Upgrade from v1 to v2](#upgrade-from-v1-to-v2) + - [Vault](#vault) + - [Insert and tokenize data: `.insert(request)`](#insert-and-tokenize-data-insertrequest) + - [Insert example with `continueOnError` option](#insert-example-with-continueonerror-option) + - [Upsert request](#upsert-request) + - [Detokenize: `.detokenize(request, options)`](#detokenize-detokenizerequest-options) + - [Construct a detokenize request](#construct-a-detokenize-request) + - [Get Record(s): `.get(request, options)`](#get-records-getrequest-options) + - [Construct a get request](#construct-a-get-request) + - [Get by Skyflow IDs](#get-by-skyflow-ids) + - [Get tokens for records](#get-tokens-for-records) + - [Get by column name and column values](#get-by-column-name-and-column-values) + - [Redaction Types](#redaction-types) + - [Update Records](#update-records) + - [Construct an update request](#construct-an-update-request) + - [Delete Records](#delete-records) + - [Query](#query) + - [Upload File](#upload-file) + - [Retrieve Existing Tokens: `.tokenize(request)`](#retrieve-existing-tokens-tokenizerequest) + - [Construct a `.tokenize()` request](#construct-a-tokenize-request) + - [Detect](#detect) + - [De-identify Text: `.deidentifyText(request, options)`](#de-identify-text-deidentifytextrequest-options) + - [Re-identify Text: `.reidentifyText(request, options)`](#re-identify-text-reidentifytextrequest-options) + - [De-identify File: `.deidentifyFile(fileReq, options)`](#de-identify-file-deidentifyfilefilereq-options) + - [Get Run: `.getDetectRun(request)`](#get-run-getdetectrunrequest) + - [Connections](#connections) + - [Configure a connection](#configure-a-connection) + - [Invoke a connection](#invoke-a-connection) + - [Construct an invoke connection request](#construct-an-invoke-connection-request) + - [Authentication \& authorization](#authentication--authorization) + - [Types of `credentials`](#types-of-credentials) + - [Generate bearer tokens for authentication \& authorization](#generate-bearer-tokens-for-authentication--authorization) + - [Generate a bearer token](#generate-a-bearer-token) + - [`generateBearerToken(filepath)`](#generatebearertokenfilepath) + - [`generateBearerTokenFromCreds(credentialsString)`](#generatebearertokenfromcredscredentialsstring) + - [Generate bearer tokens scoped to certain roles](#generate-bearer-tokens-scoped-to-certain-roles) + - [Generate bearer tokens with `ctx` for context-aware authorization](#generate-bearer-tokens-with-ctx-for-context-aware-authorization) + - [Generate signed data tokens](#generate-signed-data-tokens) + - [Client management](#client-management) + - [Using the client in production](#using-the-client-in-production) + - [Logging](#logging) + - [Example `skyflowConfig.logLevel: LogLevel.INFO`](#example-skyflowconfigloglevel-loglevelinfo) + - [Error handling](#error-handling) + - [Catching `SkyflowError` instances](#catching-skyflowerror-instances) + - [Per-record errors (`SkyflowRecordError`)](#per-record-errors-skyflowrecorderror) + - [Bearer token expiration edge cases](#bearer-token-expiration-edge-cases) + - [Troubleshooting](#troubleshooting) + - [TypeScript types reference](#typescript-types-reference) + - [Security](#security) + - [Reporting a Vulnerability](#reporting-a-vulnerability) + +## Overview + +The Skyflow SDK enables you to connect to your Skyflow Vault(s) to securely handle sensitive data at rest, in-transit, and in-use. + +> [!TIP] +> Looking for the full list of request parameters, options setters, response fields, enums, and Detect helper classes? See the **[API Reference](docs/api_reference.md)**. + +> [!IMPORTANT] +> This readme documents SDK version 2. +> For version 1 see the [v1.14.2 README](https://github.com/skyflowapi/skyflow-node/tree/1.14.2). +> For more information on how to migrate see [MIGRATE_TO_V2.md](docs/migrate_to_v2.md). + +## Installation + +Requires Node v12.22.12 and above. + +```sh +npm install skyflow-node +``` + +Depending on your project setup, you may use either the `require` method (common in Node.js projects) or the `import` statement (common in projects using ES modules). + +### Require + +```typescript +const { Skyflow } = require("skyflow-node"); +``` + +### ES modules + +```typescript +import { Skyflow } from "skyflow-node"; +``` + +### All imports + +```typescript +import { + Skyflow, // Vault client + isExpired, // JWT auth helpers + LogLevel, // logging options +} from "skyflow-node"; +``` + +## Quickstart + +Get started quickly with the essential steps: authenticate, initialize the client, and perform a basic vault operation. This section shows you a minimal working example. + +### Before you begin + +You need a Skyflow account and a few values from Skyflow Studio. If you don't have an account, [request a demo](https://www.skyflow.com/get-demo). + +| Value | Where to find it | +|---|---| +| `vaultId` | Your vault's details page in Skyflow Studio. | +| `clusterId` | The first segment of your vault URL: `https://{clusterId}.vault.skyflowapis.com`. | +| `env` | The environment your vault runs in — `Env.PROD`, `Env.SANDBOX`, `Env.DEV`, or `Env.STAGE` (defaults to `PROD`). | +| Credentials | Create a **service account** in Studio. Choose **API key** during creation for the simplest setup, or download the `credentials.json` file for token-based auth. See [Authentication & authorization](#authentication--authorization). | + +The quickstart below assumes a table named `table1` with `card_number` and `cardholder_name` columns. Adjust the table and column names to match your vault schema. See the [Skyflow docs](https://docs.skyflow.com/) for creating vaults, tables, and service accounts. + +### Authenticate + +You can use an API key or a personal bearer token to directly authenticate and authorize requests with the SDK. Use API keys for long-term service authentication. Use bearer tokens for optimal security. + +### API Key + +```typescript +import { Credentials } from 'skyflow-node'; + +const credentials: Credentials = { + apiKey: "", +}; +``` + +### Bearer Token (static) + +```typescript +import { Credentials } from 'skyflow-node'; + +const credentials: Credentials = { + token: "", +}; +``` + +For authenticating via generated bearer tokens including support for scoped tokens, context-aware access tokens, and more, refer to the [Authentication & Authorization](#authentication--authorization) section. + +### Initialize the client + +Initialize the Skyflow client first. You can specify different credential types during initialization. + +```javascript +import { Skyflow, SkyflowConfig, VaultConfig, Env, LogLevel } from 'skyflow-node'; + +// Create a credentials object. We'll use an API key. +const skyflowCredentials = { + apiKey: "" +}; + +// Configure vault +const vaultConfig: VaultConfig = { + vaultId: '', // required: vault identifier + clusterId: '', // required: from vault URL + env: Env.PROD, // optional: PROD (default), SANDBOX, DEV, STAGE + // credentials: vaultCredentials // optional: per-vault credentials; overrides skyflowCredentials +}; + +// Initialize Skyflow client +const skyflowConfig: SkyflowConfig = { + vaultConfigs: [vaultConfig], + skyflowCredentials: skyflowCredentials, + logLevel: LogLevel.ERROR +}; + +const skyflowClient: Skyflow = new Skyflow(skyflowConfig); +``` + +See [docs/advanced_initialization.md](docs/advanced_initialization.md) for advanced initialization examples including multiple vaults and different credential types. + +### Insert data into the vault, get tokens back + +Insert data into your vault using the `insert` method. Set `insertOptions.setReturnTokens(true)` to ensure values are tokenized in the response. + +Create an insert request with the [`InsertRequest`](docs/api_reference.md#insertrequest) class, which includes the values to be inserted as a list of records. + +Below is a simple example to get started. See the [Insert and tokenize data](#insert-and-tokenize-data-insertrequest) section for advanced options. + +```javascript +import { InsertRequest, InsertOptions } from "skyflow-node"; + +// Insert sensitive data into the vault +const insertData = [ + { card_number: "4111111111111112", cardholder_name: "John Doe" }, +]; +const insertReq = new InsertRequest("table1", insertData); + +const insertOptions = new InsertOptions(); +insertOptions.setReturnTokens(true); + +const insertResponse = await skyflowClient + .vault(vaultId) + .insert(insertReq, insertOptions); + +console.log("Insert response:", insertResponse); +``` + +Returns an [`InsertResponse`](docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setReturnTokens(true)`, each entry includes the `skyflowId` and a token per column: + +```text +Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '5391-4629-3722-7102', cardholder_name: '0f6b8a2c-90ab-4cde-9def-567890abcdef' }], errors: null } +``` + +## Upgrade from v1 to v2 + +Upgrade from `skyflow-node` v1 using the dedicated guide in [docs/migrate_to_v2.md](docs/migrate_to_v2.md). + +## Vault + +The [Vault](https://docs.skyflow.com/docs/vaults) performs operations on the vault such as inserting records, detokenizing tokens, retrieving tokens for list of `skyflowId`s and to invoke the Connection. + +### Insert and tokenize data: `.insert(request)` + +Pass options to the `insert` method to enable additional functionality such as returning tokenized data, upserting records, or allowing bulk operations to continue despite errors. See [Quickstart](#quickstart) for a basic example. + +```typescript +import { InsertRequest, InsertResponse } from 'skyflow-node'; + +const insertRequest = new InsertRequest('table1', [ + { + : '', + : '', + }, + { + : '', + : '', + }, +]); + +const response: InsertResponse = await skyflowClient + .vault('') + .insert(insertRequest); + +console.log('Insert response:', response); +``` + +Returns an [`InsertResponse`](docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setContinueOnError(true)`, `insertedFields` contains successful records and `errors` contains per-record failures: + +```text +Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', '': '', '': '' }], errors: null } +``` + +> **Note:** The response key is `skyflowId`. The legacy `skyflow_id` key is deprecated and will be removed in an upcoming release. + +#### Insert example with `continueOnError` option + +Set the `continueOnError` flag to `true` to allow insert operations to proceed despite encountering partial errors. + +> [!TIP] +> See the full example in the samples directory: [insert-continue-on-error.ts](samples/vault-api/insert-continue-on-error.ts) + +#### Upsert request + +Turn an insert into an 'update-or-insert' operation using the upsert option. The vault checks for an existing record with the same value in the specified column. If a match exists, the record updates; otherwise, a new record inserts. + +```typescript +// Specify the column to use as the index for the upsert. +// Note: The column must have the `unique` constraint configured in the vault. +insertOptions.setUpsertColumn("cardholder_name"); +``` + +For all `InsertOptions` setters and `TokenMode` values, see [InsertOptions](docs/api_reference.md#insertoptions) in the API reference. + +### Detokenize: `.detokenize(request, options)` + +Convert tokens back into plaintext values (or masked values) using the `.detokenize()` method. Detokenization accepts tokens and returns values. + +Create a detokenization request with the [`DetokenizeRequest`](docs/api_reference.md#detokenizerequest) class, which requires a list of tokens and column groups as input. + +Provide optional parameters such as the redaction type and the option to continue on error. + +#### Construct a detokenize request + +```typescript +import { + DetokenizeOptions, + DetokenizeRequest, + DetokenizeResponse, +} from "skyflow-node"; + +const detokenizeRequest = new DetokenizeRequest([ + { token: "token1", redactionType: RedactionType.PLAIN_TEXT }, + { token: "token2", redactionType: RedactionType.PLAIN_TEXT }, +]); + +const detokenizeOptions = new DetokenizeOptions(); +detokenizeOptions.setContinueOnError(true); +detokenizeOptions.setDownloadUrl(false); + +const response: DetokenizeResponse = await skyflowClient + .vault(primaryVaultConfig.vaultId) + .detokenize(detokenizeRequest, detokenizeOptions); + +console.log("Detokenization response:", response); +``` + +Returns a [`DetokenizeResponse`](docs/api_reference.md#detokenizeresponse) (`detokenizedFields`, `errors`): + +```text +Detokenization response: { detokenizedFields: [{ token: '5391-4629-3722-7102', value: '4111111111111112', type: 'STRING' }], errors: null } +``` + +> [!TIP] +> See the full example in the samples directory: [detokenzie-records.ts](samples/vault-api/detokenzie-records.ts) + +### Get Record(s): `.get(request, options)` + +Retrieve data using Skyflow IDs or unique column values with the get method. Create a get request with the [`GetRequest`](docs/api_reference.md#getrequest) class, specifying parameters such as the table name, redaction type, Skyflow IDs, column names, and column values. + +> [!NOTE] +> You can't use both Skyflow IDs and column name/value pairs in the same request. Use the `GetOptions` class to specify whether to return tokens. + +#### Construct a get request + +```typescript +import { GetRequest, GetOptions, GetResponse } from "skyflow-node"; + +const getRequest = new GetRequest("table1", ["", ""]); + +const getOptions = new GetOptions(); +getOptions.setReturnTokens(false); + +const response: GetResponse = await skyflowClient + .vault("") + .get(getRequest, getOptions); + +console.log("Get response:", response); +``` + +Returns a [`GetResponse`](docs/api_reference.md#getresponse) (`data`, `errors`): + +```text +Get response: { data: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '4111111111111112', cardholder_name: 'John Doe' }], errors: null } +``` + +> **Note:** The response key is `skyflowId`. The legacy `skyflow_id` key is deprecated and will be removed in an upcoming release. + +#### Get by Skyflow IDs + +Retrieve specific records using Skyflow IDs. Use this method when you know the exact record IDs. + +```typescript +import { + GetOptions, + GetRequest, + SkyflowError, + GetResponse, + RedactionType, +} from "skyflow-node"; + +// Initialize a list of Skyflow IDs to retrieve records (replace with actual Skyflow IDs) +const getIds: Array = [ + "a581d205-1969-4350-acbe-a2a13eb871a6", + "5ff887c3-b334-4294-9acc-70e78ae5164a", +]; + +// Step 2: Create a GetRequest to retrieve records by Skyflow ID +const getRequest: GetRequest = new GetRequest( + "table1", // Replace with your actual table name + getIds, +); + +// Step 3: Configure Get Options and specify not to return tokens and redaction type +const getOptions: GetOptions = new GetOptions(); +getOptions.setReturnTokens(false); // Optional: Set to false to avoid returning tokens +getOptions.setRedactionType(RedactionType.PLAIN_TEXT); + +// Step 4: Send the request to the Skyflow vault and retrieve the records +const getResponse: GetResponse = await skyflowClient + .vault(primaryVaultConfig.vaultId) + .get(getRequest, getOptions); +// Replace with your actual Skyflow vault ID + +console.log("Data retrieval successful:", getResponse); +``` + +#### Get tokens for records + +Return tokens for records to securely process sensitive data while maintaining data privacy. + +```ts +getOptions.setReturnTokens(true); // Set to `true` to get tokens +``` + +> [!TIP] +> See the full example in the samples directory: [get-records.ts](samples/vault-api/get-records.ts) + +#### Get by column name and column values + +Retrieve records by unique column values when you don't know the Skyflow IDs. Use this method to query data with alternate unique identifiers. + +```ts +const getRequest: GetColumnRequest = new GetColumnRequest( + tableName, + columnName, + columnValues, // Column values of the records to return +); +``` + +> [!TIP] +> See the full example in the samples directory: [get-column-values.ts](samples/vault-api/get-column-values.ts) + +#### Redaction Types + +Use redaction types to control how sensitive data displays when retrieved from the vault. + +**Available Redaction Types** + +- `DEFAULT`: Applies the vault-configured default redaction setting. +- `REDACTED`: Completely removes sensitive data from view. +- `MASKED`: Partially obscures sensitive information. +- `PLAIN_TEXT`: Displays the full, unmasked data. + +**Choosing the Right Redaction Type** + +- Use `REDACTED` for scenarios requiring maximum data protection to prevent exposure of sensitive information. +- Use `MASKED` to provide partial visibility of sensitive data for less critical use cases. +- Use `PLAIN_TEXT` for internal, authorized access where full data visibility is necessary. + +For all `GetOptions` setters and `OrderByEnum` values, see [GetOptions](docs/api_reference.md#getoptions) in the API reference. + +### Update Records + +Update data in your vault using the `update` method. Create an update request with the [`UpdateRequest`](docs/api_reference.md#updaterequest) class, specifying parameters such as the table name and data (as a dictionary). + +Configure update options using the `UpdateOptions` class to control returnTokens, tokens, and tokenMode. When `returnTokens` is `true`, Skyflow returns tokens for the updated records. When `false`, Skyflow returns IDs for the updated records. + +#### Construct an update request + +```typescript +import { UpdateRequest, UpdateOptions, UpdateResponse, TokenMode } from 'skyflow-node'; + +const updateRequest = new UpdateRequest('table1', { + skyflowId: '', + : '', + : '' +}); + +const updateOptions = new UpdateOptions(); +updateOptions.setReturnTokens(true); // return tokens for updated fields + +const response: UpdateResponse = await skyflowClient + .vault('') + .update(updateRequest, updateOptions); + +console.log('Update response:', response); +``` + +Returns an [`UpdateResponse`](docs/api_reference.md#updateresponse) (`updatedField`, `errors`). With the default `setReturnTokens(false)`, `updatedField` contains only `skyflowId`: + +```text +Update response: { updatedField: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c' }, errors: null } +``` + +> **Note:** The response key is `skyflowId`. The legacy `skyflow_id` key is deprecated and will be removed in an upcoming release. + +For all `UpdateOptions` setters and `TokenMode` values, see [UpdateOptions](docs/api_reference.md#updateoptions) in the API reference. + +> [!TIP] +> See the full example in the samples directory: [update-record.ts](samples/vault-api/update-record.ts) + +### Delete Records + +Delete records using Skyflow IDs with the `delete` method. Create a delete request with the [`DeleteRequest`](docs/api_reference.md#deleterequest) class, which accepts a list of Skyflow IDs: + +```typescript +import { DeleteRequest, DeleteResponse } from "skyflow-node"; + +const deleteRequest = new DeleteRequest("table1", [ + "", + "", + "", +]); + +const response: DeleteResponse = await skyflowClient + .vault("") + .delete(deleteRequest); + +console.log("Delete response:", response); +``` + +Returns a [`DeleteResponse`](docs/api_reference.md#deleteresponse) (`deletedIds`, `errors`): + +```text +Delete response: { deletedIds: ['a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c'], errors: null } +``` + +> [!TIP] +> See the full example in the samples directory: [delete-records.ts](samples/vault-api/delete-records.ts) + +### Query + +Retrieve data with SQL queries using the `query` method. Create a query request with the [`QueryRequest`](docs/api_reference.md#queryrequest) class, which takes the `query` parameter as follows: + + + +```typescript +import { QueryRequest, QueryResponse } from "skyflow-node"; + +const queryRequest = new QueryRequest( + "SELECT * FROM table1 WHERE column1 = 'value'", +); + +const response: QueryResponse = await skyflowClient + .vault("") + .query(queryRequest); + +console.log("Query response:", response); +``` + +Returns a [`QueryResponse`](docs/api_reference.md#queryresponse) (`fields`, `errors`). Each record also includes a `tokenizedData` map: + +```text +Query response: { fields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '4111111111111112', tokenizedData: { card_number: '5391-4629-3722-7102' } }], errors: null } +``` + +> [!TIP] +> See the full example in the samples directory: [query-records.ts](samples/vault-api/query-records.ts) + +Refer to [Query your data](https://docs.skyflow.com/query-data/) and [Execute Query](https://docs.skyflow.com/record/#QueryService_ExecuteQuery) for guidelines and restrictions on supported SQL statements, operators, and keywords. + +### Upload File + +Upload files to a Skyflow vault using the `uploadFile` method. Requires Node.js v20 or above. Create a [`FileUploadRequest`](docs/api_reference.md#fileuploadrequest) with the table name and column name, then configure a `FileUploadOptions` with the file source and optionally a Skyflow ID. + +**Attach a file to an existing record:** + +```typescript +import { FileUploadRequest, FileUploadResponse, FileUploadOptions } from "skyflow-node"; + +const uploadReq = new FileUploadRequest("", ""); + +const uploadOptions = new FileUploadOptions(); +uploadOptions.setSkyflowId(""); // attach to this record +uploadOptions.setFilePath("path/to/file.pdf"); + +const response: FileUploadResponse = await skyflowClient + .vault("") + .uploadFile(uploadReq, uploadOptions); + +console.log("File upload:", response); +``` + +**Upload a file and create a new record (omit `setSkyflowId`):** + +```typescript +const uploadReq = new FileUploadRequest("documents", "attachment"); + +const uploadOptions = new FileUploadOptions(); +uploadOptions.setFilePath("path/to/file.pdf"); // no setSkyflowId — creates a new record + +const response: FileUploadResponse = await skyflowClient + .vault("") + .uploadFile(uploadReq, uploadOptions); + +console.log("File upload:", response); +``` + +Both forms return a [`FileUploadResponse`](docs/api_reference.md#fileuploadresponse) (`skyflowId`, `errors`): + +```text +File upload: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', errors: null } +``` + +For all `FileUploadOptions` setters, see [FileUploadOptions](docs/api_reference.md#fileuploadoptions) in the API reference. + +> [!TIP] +> See the full example in the samples directory: [file-upload.ts](samples/vault-api/file-upload.ts) + +### Retrieve Existing Tokens: `.tokenize(request)` + +Retrieve tokens for values that already exist in the vault using the `.tokenize()` method. This method returns existing tokens only and does not generate new tokens. Build the request with the [`TokenizeRequest`](docs/api_reference.md#tokenizerequest) class. + +#### Construct a `.tokenize()` request + +```typescript +import { TokenizeRequest, TokenizeResponse } from "skyflow-node"; + +const tokenizeRequest = new TokenizeRequest([ + { value: "", columnGroup: "" }, + { value: "", columnGroup: "" }, +]); + +const response: TokenizeResponse = await skyflowClient + .vault("") + .tokenize(tokenizeRequest); + +console.log("Tokenization Result:", response); +``` + +Returns a [`TokenizeResponse`](docs/api_reference.md#tokenizeresponse) (`tokens`, `errors`): + +```text +Tokenization Result: { tokens: ['5391-4629-3722-7102', '0f6b8a2c-90ab-4cde-9def-567890abcdef'], errors: null } +``` + +> [!TIP] +> See the full example in the samples directory: [tokenize-records.ts](samples/vault-api/tokenize-records.ts) + +## Detect + +De-identify and reidentify sensitive data in text and files using Skyflow Detect, which supports advanced privacy-preserving workflows. + +### De-identify Text: `.deidentifyText(request, options)` + +De-identify or anonymize text using the `deidentifyText` method. + +Create a de-identify text request with the [`DeidentifyTextRequest`](docs/api_reference.md#deidentifytextrequest) class, which includes the text to be deidentified. Provide optional parameters using the [`DeidentifyTextOptions`](docs/api_reference.md#deidentifytextoptions) class. + +```typescript +import { + DeidentifyTextRequest, + DeidentifyTextOptions, + SkyflowError, + TokenFormat, + TokenType, + Transformations, + DetectEntities, +} from "skyflow-node"; + +// Prepare the text to be deidentified +const deidentifyTextRequest = new DeidentifyTextRequest( + "", +); + +// Configure DeidentifyTextOptions +const options = new DeidentifyTextOptions(); +options.setEntities([DetectEntities.ACCOUNT_NUMBER, DetectEntities.SSN]); // Entities to de-identify +options.setAllowRegexList([""]); // Allowlist regex patterns +options.setRestrictRegexList([""]); // Restrict regex patterns + +const tokenFormat = new TokenFormat(); +tokenFormat.setDefault(TokenType.VAULT_TOKEN); // default format +options.setTokenFormat(tokenFormat); + +const transformations = new Transformations(); +transformations.setShiftDays({ + max: 30, // maximum days to shift + min: 10, // minimum days to shift + entities: [DetectEntities.ACCOUNT_NUMBER, DetectEntities.SSN], // entity types to apply the shift +}); +options.setTransformations(transformations); + +// Call deidentifyText +const response = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyText(deidentifyTextRequest, options); + +console.log("De-identify Text Response:", response); +``` + +Returns a [`DeidentifyTextResponse`](docs/api_reference.md#deidentifytextresponse) (`processedText`, `entities`, `wordCount`, `charCount`, `errors`): + +```text +De-identify Text Response: { processedText: 'Hello, [PERSON_1]! My SSN is [SSN_1].', entities: [{ token: '[PERSON_1]', value: 'John', entity: 'NAME', scores: { NAME: 0.98 } }], wordCount: 8, charCount: 37, errors: null } +``` + +> [!TIP] +> See the full example in the samples directory: [deidentify-text.ts](samples/detect-api/deidentify-text.ts) + +### Re-identify Text: `.reidentifyText(request, options)` + +Re-identify text using the `reidentifyText` method. Create a reidentify text request with the [`ReidentifyTextRequest`](docs/api_reference.md#reidentifytextrequest) class, which includes the redacted or de-identified text to be re-identified. Provide optional parameters using the [`ReidentifyTextOptions`](docs/api_reference.md#reidentifytextoptions) class to control how specific entities are returned (as redacted, masked, or plain text). + +```typescript +import { + ReidentifyTextRequest, + ReidentifyTextOptions, + SkyflowError, + DetectEntities, + ReidentifyTextResponse, +} from "skyflow-node"; + +// Prepare the redacted text to be re-identified +const request = new ReidentifyTextRequest(""); + +// Configure ReidentifyTextOptions +const options = new ReidentifyTextOptions(); +options.setRedactedEntities([DetectEntities.SSN]); // Entities to keep redacted +options.setMaskedEntities([DetectEntities.CREDIT_CARD_NUMBER]); // Entities to mask +options.setPlainTextEntities([DetectEntities.NAME]); // Entities to return as plain text + +// Call reidentifyText +const response: ReidentifyTextResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyText(request, options); + +console.log("Reidentify Text Response:", response); +``` + +Returns a [`ReidentifyTextResponse`](docs/api_reference.md#reidentifytextresponse) (`processedText`): + +```text +Reidentify Text Response: { processedText: 'Hello, John! My SSN is 123-45-6789.' } +``` + +> [!TIP] +> See the full example in the samples directory: [reidentify-text.ts](samples/detect-api/reidentify-text.ts) + +### De-identify File: `.deidentifyFile(fileReq, options)` + +De-identify files using the `deidentifyFile` method. Create a de-identify file request with the [`DeidentifyFileRequest`](docs/api_reference.md#deidentifyfilerequest) class, which includes the file to be deidentified (such as images, PDFs, audio, documents, spreadsheets, or presentations). Provide optional parameters using the [`DeidentifyFileOptions`](docs/api_reference.md#deidentifyfileoptions) class to control how entities are detected and deidentified, as well as how the output is generated for different file types. + +> [!NOTE] +> File de-identification requires Node.js v20.x or above. + +```typescript +import { + DeidentifyFileRequest, + DeidentifyFileOptions, + DeidentifyFileResponse + SkyflowError, + DetectEntities, + MaskingMethod, + DetectOutputTranscription, + Bleep +} from 'skyflow-node'; + +// Prepare the file to be deidentified +const filePath: string = ''; +const buffer = fs.readFileSync(filePath); +const file = new File([buffer], filePath); + +// Construct the file input by providing either a file object or file path, but not both +const fileInput: FileInput = { file: file } // OR const fileInput: FileInput = { filePath: filePath } +const fileReq = new DeidentifyFileRequest(fileInput); + +// Configure DeidentifyFileOptions +const options = new DeidentifyFileOptions(); +options.setEntities([DetectEntities.SSN, DetectEntities.ACCOUNT_NUMBER]); + +const tokenFormat = new TokenFormat(); // Token format for deidentified entities +tokenFormat.setDefault(TokenType.ENTITY_ONLY); +options.setTokenFormat(tokenFormat); + + +options.setOutputDirectory(''); // Output directory for saving the deidentified file. Not supported in Cloudflare workers. +options.setWaitTime(64); // Wait time for polling (max 64 seconds; returns runId + status if exceeded) + + +tifyFile +const response: DeidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(fileReq, options); + +console.log('De-identify File Response:', response); +``` + +Returns a [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) (`fileBase64`, `file`, `runId`, `status`, `errors`). If processing exceeds `setWaitTime`, only `runId` and `status` are populated — poll with `getDetectRun`: + +```text +De-identify File Response: { runId: 'run-abc123', status: 'IN_PROGRESS', fileBase64: undefined, errors: null } +``` + +**Supported file types:** + +- Documents: `doc`, `docx`, `pdf` +- PDFs: `pdf` +- Images: `bmp`, `jpeg`, `jpg`, `png`, `tif`, `tiff` +- Structured text: `json`, `xml` +- Spreadsheets: `csv`, `xls`, `xlsx` +- Presentations: `ppt`, `pptx` +- Audio: `mp3`, `wav` + +**Notes:** +- Transformations can't be applied to Documents, Images, or PDFs. +- `setWaitTime` must be ≤ 64 seconds; otherwise, an error is thrown. +- If the API takes more than `waitTime` seconds, the response contains only `runId` and `status`. Use `.getDetectRun()` to poll for the result. + +For all `DeidentifyFileOptions` setters and the `Bleep` class, see [DeidentifyFileOptions](docs/api_reference.md#deidentifyfileoptions) in the API reference. + +> [!TIP] +> See the full example in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) + +### Get Run: `.getDetectRun(request)` + +Retrieve the results of a previously started file de-identification operation (or 'run') using the `getDetectRun(...)` method. Initialize the [`GetDetectRunRequest`](docs/api_reference.md#getdetectrunrequest) class with the `runId` returned from a prior `.deidentifyFile(fileReq, options)` call. Fetch the final results of the file de-identification operation once they are available. + +```typescript +import { + GetDetectRunRequest, + DeidentifyFileResponse, + DeidentifyFileResponse + SkyflowError +} from 'skyflow-node'; + +// Prepare the GetDetectRunRequest with the runId from a previous deidentifyFile call +const request = new GetDetectRunRequest({ + runId: '', // Replace with the runId you received earlier +}); + +// Step 2: Call getDetectRun +const response: DeidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .getDetectRun(request); + +// Step 3: Handle the response +console.log('Get Detect Run Response:', response); +``` + +Returns a [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) with the current `status` (and the processed file once complete): + +```text +Get Detect Run Response: { status: 'SUCCESS', runId: 'run-abc123', fileBase64: '', errors: null } +``` + +> [!TIP] +> See the full example in the samples directory: [get-detect-run.ts](samples/detect-api/get-detect-run.ts) + +## Connections + +Securely send and receive data between your systems and first- or third-party services using Skyflow Connections, a gateway service that uses tokenization. The [connections](https://github.com/skyflowapi/skyflow-node/tree/v2/src/vault/controller/connections) module invokes both inbound and outbound connections. + +- **Inbound connections**: Act as intermediaries between your client and server, tokenizing sensitive data before it reaches your backend, ensuring downstream services handle only tokenized data. +- **Outbound connections**: Enable secure extraction of data from the vault and transfer it to third-party services via your backend server, such as processing checkout or card issuance flows. + +### Configure a connection + +Add a `ConnectionConfig` to `SkyflowConfig.connectionConfigs` during initialization, or call `addConnectionConfig()` at runtime: + +```typescript +import { ConnectionConfig, Credentials } from 'skyflow-node'; + +const connectionConfig: ConnectionConfig = { + connectionId: '', // required: unique connection identifier + connectionUrl: '', // required: gateway URL for this connection + // credentials: connectionCredentials // optional: per-connection credentials; overrides skyflowCredentials +}; +``` + +### Invoke a connection + +Invoke a connection using the `invoke` method of the Skyflow client. Build the request with the [`InvokeConnectionRequest`](docs/api_reference.md#invokeconnectionrequest) class. + +#### Construct an invoke connection request + +```typescript +import { + InvokeConnectionRequest, + RequestMethod, + InvokeConnectionResponse, +} from "skyflow-node"; + +const invokeRequest = new InvokeConnectionRequest( + RequestMethod.POST, + { : "" }, + { : "" }, + { : "" }, + { : "" } +); + +const response: InvokeConnectionResponse = await skyflowClient + .connection() + .invoke(invokeRequest); + +console.log("Invoke connection response:", response); +``` + +Returns an [`InvokeConnectionResponse`](docs/api_reference.md#invokeconnectionresponse) (`data`, `metadata`, `errors`): + +```text +Invoke connection response: { data: { status: 'success', transaction_id: 'txn_abc123' }, metadata: { requestId: 'req-xyz-456' }, errors: null } +``` + +`method` supports the following methods (see [`RequestMethod`](docs/api_reference.md#requestmethod)): + +- `GET` +- `POST` +- `PUT` +- `PATCH` + +**pathParams, queryParams, header, body** are the JSON objects represented as dictionaries that will be sent through the connection integration url. + +> [!TIP] +> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/vault-api/invoke-connection.ts) +> See [docs.skyflow.com](https://docs.skyflow.com) for more details on integrations with Connections, Functions, and Pipelines. + +## Authentication & authorization + +### Types of `credentials` + +The SDK accepts one of several types of credentials object. + +1. **API keys** + A unique identifier used to authenticate and authorize requests to an API. Use for long-term service authentication. To create an API key, first create a 'Service Account' in Skyflow and choose the 'API key' option during creation. + + ```ts + const credentials: Credentials = { + apiKey: "" + }; + ``` + +2. **Bearer tokens** + A temporary access token used to authenticate API requests. Use for optimal security. As a developer with the right access, you can generate a temporary personal bearer token in Skyflow in the user menu. + + ```ts + const credentials: Credentials = { + token: "" + }; + ``` + +3. **Service account credentials file path** + The file path pointing to a JSON file containing credentials for a service account. Use when credentials are managed externally or stored in secure file systems. + + ```ts + const credentials: Credentials = { + path: "" + }; + ``` + +4. **Service account credentials string** + JSON-formatted string containing service account credentials. Use when integrating with secret management systems or when credentials are passed programmatically. + + ```ts + const credentials: Credentials = { + credentialsString: JSON.stringify(process.env.SKYFLOW_CREDENTIALS) + }; + ``` + +5. **Environment variables** + If no credentials are explicitly provided, the SDK automatically looks for the SKYFLOW_CREDENTIALS environment variable. Use to avoid hardcoding credentials in source code. This variable must return an object like one of the examples above. + +> [!NOTE] +> Only one type of credential can be used at a time. If multiple credentials are provided, the last one added will take precedence. + +### Generate bearer tokens for authentication & authorization + +Generate and manage bearer tokens to authenticate API calls. This section covers options for scoping to certain roles, passing context, and signing data tokens. + +#### Generate a bearer token + +Generate service account tokens using the [Service Account](https://github.com/skyflowapi/skyflow-node/tree/v2/src/service-account) Node package with a service account credentials file provided when a service account is created. Tokens generated by this module are valid for 60 minutes and can be used to make API calls to the [Data](https://docs.skyflow.com/record/) and [Management](https://docs.skyflow.com/management/) APIs, depending on the permissions assigned to the service account. + +##### `generateBearerToken(filepath)` + +The `generateBearerToken(filepath)` function takes the `credentials.json` file path for token generation. + +```ts +import { generateBearerToken, BearerTokenOptions } from 'skyflow-node'; + +const options: BearerTokenOptions = { + roleIds: ['roleId1'], // optional: scope to specific roles + ctx: 'user_12345', // optional: embed context + logLevel: LogLevel.ERROR, +}; + +generateBearerToken('path/to/credentials.json', options) + .then(response => { + const bearerToken = response.accessToken; + }) + .catch(error => { /* handle error */ }); +``` + +##### `generateBearerTokenFromCreds(credentialsString)` + +Use `generateBearerTokenFromCreds` when your credentials JSON is available as a string (e.g. from a secret manager) rather than a file on disk. It accepts the same `BearerTokenOptions` as `generateBearerToken`. + +```ts +import { generateBearerTokenFromCreds, BearerTokenOptions } from 'skyflow-node'; + +const credentialsString = JSON.stringify(require('./credentials.json')); +const options: BearerTokenOptions = { roleIds: ['roleId1'] }; + +const response = await generateBearerTokenFromCreds(credentialsString, options); +const bearerToken = response.accessToken; +``` + +**`BearerTokenOptions` type:** + +```ts +type BearerTokenOptions = { + roleIds?: string[]; // Scope the token to specific service-account roles + ctx?: string | Record; // Context value(s) embedded in the token + logLevel?: LogLevel; // Override SDK log level for this call + tokenUri?: string; // Override the token endpoint URL from the credentials file +}; +``` + + +**Checking whether a token has expired — `isExpired`:** + +```ts +import { isExpired } from 'skyflow-node'; + +// Returns true if the JWT bearer token is expired or will expire within the threshold +const expired: boolean = isExpired(bearerToken); +if (expired) { + // regenerate before making API calls + const { accessToken } = await generateBearerToken('path/to/credentials.json'); +} +``` + +> [!TIP] +> See the full example in the samples directory: [token-generation-example.ts](http://github.com/skyflowapi/skyflow-node/blob/v2/samples/service-account/token-generation-example.ts) + +#### Generate bearer tokens scoped to certain roles + +Generate bearer tokens with access limited to a specific role by specifying the appropriate `roleIds` when using a service account with multiple roles. Use this to limit access for services with multiple responsibilities, such as segregating access for billing and analytics. Generated bearer tokens are valid for 60 minutes and can only execute operations permitted by the permissions associated with the designated role. + +```ts +const options: BearerTokenOptions = { + roleIds: ['roleId1', 'roleId2'], +}; +``` + +> [!TIP] +> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/service-account/scoped-token-generation-example.ts) +> See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. + +#### Generate bearer tokens with `ctx` for context-aware authorization + +Embed context values into a bearer token during generation so you can reference those values in your policies. This enables more flexible access controls, such as tracking end-user identity when making API calls using service accounts, and facilitates using signed data tokens during detokenization. + +Generate bearer tokens containing context information using a service account with the `context_id` identifier. Context information is represented as a JWT claim in a Skyflow-generated bearer token. Tokens generated from such service accounts include a `context_identifier` claim, are valid for 60 minutes, and can be used to make API calls to the Data and Management APIs, depending on the service account's permissions. + +The `ctx` parameter accepts either a **string** or a **JSON object**: + +**String context** — use when your policy references a single context value: + +```typescript +const options = { + ctx: 'user_12345', +}; +const response = await generateBearerToken(filepath, options); +``` + +**JSON object context** — use when your policy needs multiple context values for conditional data access. Each key in the `ctx` object maps to a Skyflow CEL policy variable under `request.context.*`: + +```typescript +const options = { + ctx: { + role: 'admin', + department: 'finance', + user_id: 'user_12345', + }, +}; +const response = await generateBearerToken(filepath, options); +``` + +With the object above, your Skyflow policies can reference `request.context.role`, `request.context.department`, and `request.context.user_id` to make conditional access decisions. + +You can also set the `context` field on credentials for automatic token generation: + +```typescript +// String context on credentials +const credentials: PathCredentials = { + path: 'path/to/credentials.json', + context: 'user_12345', +}; + +// JSON object context on credentials +const credentials: PathCredentials = { + path: 'path/to/credentials.json', + context: { + role: 'admin', + department: 'finance', + }, +}; +``` + +> [!TIP] +> See the full example in the samples directory: [token-generation-with-context-example.ts](samples/service-account/token-generation-with-context-example.ts) +> See Skyflow's [context-aware authorization](https://docs.skyflow.com) and [conditional data access](https://docs.skyflow.com) docs for policy variable syntax like `request.context.*`. + +#### Generate signed data tokens + +Digitally sign data tokens with a service account's private key to add an extra layer of protection. Skyflow generates data tokens when sensitive data is inserted into the vault. Detokenize signed tokens only by providing the signed data token along with a bearer token generated from the service account's credentials. The service account must have the necessary permissions and context to successfully detokenize the signed data tokens. + +Two variants are available — choose based on how you supply credentials: + +```ts +import { + generateSignedDataTokens, + generateSignedDataTokensFromCreds, + SignedDataTokensOptions, +} from 'skyflow-node'; + +const options: SignedDataTokensOptions = { + dataTokens: ['dataToken1', 'dataToken2'], // required: tokens to sign + timeToLive: 90, // optional: seconds until expiry (default 60) + ctx: 'user_12345', // optional: context embedded in token +}; + +// From a credentials file on disk: +const response = await generateSignedDataTokens('path/to/credentials.json', options); + +// From a credentials JSON string (e.g. from a secret manager): +const credentialsString = JSON.stringify(require('./credentials.json')); +const response = await generateSignedDataTokensFromCreds(credentialsString, options); + +response.forEach(({ token, signedToken }) => { + console.log('Original token:', token); + console.log('Signed token:', signedToken); +}); +``` + +The `ctx` parameter also accepts a JSON object for multi-value context: + +```typescript +const options: SignedDataTokensOptions = { + dataTokens: ['dataToken1'], + ctx: { role: 'analyst', department: 'research' }, +}; +``` + +**`SignedDataTokensOptions` type:** + +```ts +type SignedDataTokensOptions = { + dataTokens: string[]; // Data tokens to sign (required) + timeToLive?: number; // Seconds until token expiry (default 60) + ctx?: string | Record; // Context value(s) embedded in the token + logLevel?: LogLevel; // Override SDK log level for this call + tokenUri?: string; // Override the token endpoint URL from credentials +}; +``` + +> [!TIP] +> See the full example in the samples directory: [signed-token-generation-example.ts](samples/service-account/signed-token-generation-example.ts) +> See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. + +## Client management + +After initializing the `Skyflow` client, you can add, update, retrieve, and remove vault and connection configurations at any time without recreating the client. For the full list of methods, see [Client management methods](docs/api_reference.md#client-management-methods) in the API reference. + +## Using the client in production + +**Initialize once and reuse.** `new Skyflow(config)` returns a long-lived client that caches tokens per vault. Construct it once at startup (e.g. as a module-level singleton) and reuse it across requests. Recreating the client on every call discards these caches and forces unnecessary token regeneration. + +```ts +// At application startup +const skyflowClient = new Skyflow(skyflowConfig); + +// Reuse `skyflowClient` for the lifetime of the process +``` + +**Bearer token refresh is automatic.** When you authenticate with a service-account credentials file/string (or API key), the SDK caches the generated bearer token and regenerates it automatically once it expires. You don't need to manage token lifecycle yourself. (For the rare expire-mid-request case, see [Bearer token expiration edge cases](#bearer-token-expiration-edge-cases).) + +**Runtime configuration changes are not concurrency-safe.** Methods that mutate client state at runtime — `addVaultConfig`, `updateVaultConfig`, `removeVaultConfig`, the `*ConnectionConfig` methods, and `updateSkyflowCredentials` — should be called during setup, not concurrently with in-flight requests. Once configured, the client is safe to reuse across concurrent calls. + +**Timeouts and retries.** The SDK does not currently expose request timeout or automatic-retry configuration. Wrap your SDK calls with your own timeout/retry logic at the application layer if needed. + +## Logging + +The SDK provides useful logging. By default, the logging level is set to `LogLevel.ERROR`. Change this by setting the `logLevel` in Skyflow Config while creating the Skyflow Client as shown below: + +Currently, the following five log levels are supported: + +- `DEBUG`: + When `LogLevel.DEBUG` is passed, logs at all levels will be printed (DEBUG, INFO, WARN, ERROR). +- `INFO`: + When `LogLevel.INFO` is passed, INFO logs for every event that occurs during SDK flow execution will be printed, along with WARN and ERROR logs. +- `WARN`: + When `LogLevel.WARN` is passed, only WARN and ERROR logs will be printed. +- `ERROR`: + When `LogLevel.ERROR` is passed, only ERROR logs will be printed. +- `OFF`: + `LogLevel.OFF` can be used to turn off all logging from the Skyflow Node SDK. + +> [!NOTE] +> The ranking of logging levels is as follows: `DEBUG` < `INFO` < `WARN` < `ERROR` < `OFF`. + +### Example `skyflowConfig.logLevel: LogLevel.INFO` + +```ts +const skyflowConfig: SkyflowConfig = { + vaultConfigs: [vaultConfig], // Add the Vault configuration + skyflowCredentials: skyflowCredentials, // Use Skyflow credentials if no token is passed + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. +}; + +const skyflowClient: Skyflow = new Skyflow(skyflowConfig); +``` + +## Error handling + +### Catching `SkyflowError` instances + +Wrap your calls to the Skyflow SDK in try/catch blocks as a best practice. Use the `SkyflowError` type to identify errors coming from Skyflow versus general request/response errors. + +```ts +try { + // ...call the Skyflow SDK +} catch (error) { + // catch an error, identify if it is a SkyflowError + if (error instanceof SkyflowError) { + console.error("Skyflow Specific Error:", { + code: error.error?.httpCode, + message: error.message, + requestId: error.error?.requestId, + details: error.error?.details, + }); + } else { + console.error("Unexpected Error:", JSON.stringify(error)); + } +} +``` + +### Per-record errors (`SkyflowRecordError`) + +Some operations — such as `insert` with `continueOnError: true` and `detokenize` — can partially succeed. In these cases the response object contains an `errors` array alongside the successful results. Each entry in `errors` is a `SkyflowRecordError`: + +```ts +import { InsertResponse, SkyflowRecordError } from 'skyflow-node'; + +const response: InsertResponse = await skyflowClient.vault(vaultId).insert(request, options); + +if (response.errors && response.errors.length > 0) { + response.errors.forEach((err: SkyflowRecordError) => { + console.error({ + requestId: err.requestId, // x-request-id from the server response + httpCode: err.httpCode, // HTTP status code for this record + requestIndex: err.requestIndex, // Index of the failed record in the input array + error: err.error, // Error description string + token: err.token, // Token (detokenize errors only) + }); + }); +} +``` + +`SkyflowRecordError` shape: + +```ts +interface SkyflowRecordError { + error: string; + requestId: string | null; + httpCode?: string | number | null; + requestIndex?: number | null; + token?: string | null; // present on detokenize errors +} +``` + +### Bearer token expiration edge cases + +When using bearer tokens for authentication and API requests, a token may expire after verification but before the actual API call completes. This causes the request to fail unexpectedly. An error from this edge case looks like this: + +```txt +message: Authentication failed. Bearer token is expired. Use a valid bearer token. See https://docs.skyflow.com/api-authentication/ +``` + +If you encounter this kind of error, retry the request. During the retry the SDK detects that the previous bearer token has expired and generates a new one for the current and subsequent requests. + +> [!TIP] +> See the full example in the samples directory: [bearer-token-expiry-example.ts](samples/service-account/bearer-token-expiry-example.ts) +> See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. + +## Troubleshooting + +Most first-run problems come from configuration mismatches. Every error thrown by the SDK is a `SkyflowError` exposing `httpCode`, `message`, and `details` — inspect these first (see [Error handling](#error-handling)). + +| Symptom | Likely cause | Fix | +|---|---|---| +| `npm install skyflow-node` fails | Unsupported Node version | Requires Node v12.22.12 or above. | +| Connection/DNS failures, or 404 on every call | Wrong `clusterId` | `clusterId` is the first segment of your vault URL: `https://{clusterId}.vault.skyflowapis.com`. | +| Requests hit the wrong host / unexpected auth failures | Wrong `env` | Match `env` to where your vault runs (`Env.PROD`, `Env.SANDBOX`, `Env.DEV`, `Env.STAGE`). | +| `401 Unauthorized` | Invalid or expired credentials | Verify your API key or service-account credentials. Regenerate if needed. | +| `403 Forbidden` | Service account lacks permission | Grant the service account a role with the required permissions, or use a [scoped token](#generate-bearer-tokens-scoped-to-certain-roles). | +| `404` referencing a table or column | Name mismatch | Confirm table and column names match your vault schema exactly (case-sensitive). | +| Vault not found / 404 with a valid `clusterId` | Wrong `vaultId` | Copy `vaultId` from the vault's details page in Skyflow Studio. | +| `Authentication failed. Bearer token is expired.` | Token expired mid-request | Retry the request; the SDK regenerates the token. See [Bearer token expiration edge cases](#bearer-token-expiration-edge-cases). | +| Unexpected credential is used | Multiple credential types provided | Only one credential type is used at a time; the last one added takes precedence. | + +If you're stuck, set `logLevel: LogLevel.DEBUG` during development for detailed SDK logs (see [Logging](#logging)). + +## TypeScript types reference + +For the full reference of all exported types, interfaces, enums, request/response fields, and options setters, see the **[API Reference](docs/api_reference.md)**. + +The most commonly imported TypeScript types are: + +```ts +import { + // Credential sub-types (for stricter typing) + ApiKeyCredentials, + TokenCredentials, + PathCredentials, // supports optional tokenUri + StringCredentials, // supports optional tokenUri + + // Request item shapes + DetokenizeData, // { token: string; redactionType?: RedactionType } + TokenizeRequestType, // { value: string; columnGroup: string } + FileInput, // { file: File } | { filePath: string } + + // Response record shapes + InsertResponseType, // { skyflowId: string; [field: string]: unknown } + GetResponseData, // { [field: string]: unknown } + QueryResponseType, // { [field: string]: unknown } + SkyflowRecordError, // shape of entries in response.errors[] +} from 'skyflow-node'; +``` + +## Security +### Reporting a Vulnerability + +If you discover a potential security issue in this project, reach out to us at [security@skyflow.com](mailto:security@skyflow.com). + +Don't create public GitHub issues or Pull Requests, as malicious actors could potentially view them. diff --git a/skyvault/api-extractor.json b/skyvault/api-extractor.json new file mode 100644 index 00000000..0f7db7c9 --- /dev/null +++ b/skyvault/api-extractor.json @@ -0,0 +1,32 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json", + "projectFolder": ".", + "mainEntryPointFilePath": "/dist/index.d.ts", + "bundledPackages": [], + "compiler": { + "tsconfigFilePath": "/tsconfig.dts.json" + }, + "apiReport": { + "enabled": true, + "reportFolder": "/api-report/", + "reportTempFolder": "/temp/", + "reportFileName": ".api.md" + }, + "docModel": { "enabled": false }, + "dtsRollup": { "enabled": false }, + "tsdocMetadata": { "enabled": false }, + "messages": { + "compilerMessageReporting": { + "default": { "logLevel": "warning" } + }, + "extractorMessageReporting": { + "default": { "logLevel": "warning" }, + "ae-missing-release-tag": { "logLevel": "none" }, + "ae-setter-with-docs": { "logLevel": "none" }, + "ae-forgotten-export": { "logLevel": "none" } + }, + "tsdocMessageReporting": { + "default": { "logLevel": "none" } + } + } +} diff --git a/api-report/skyflow-node.api.md b/skyvault/api-report/skyflow-node.api.md similarity index 59% rename from api-report/skyflow-node.api.md rename to skyvault/api-report/skyflow-node.api.md index b22142fa..39b00709 100644 --- a/api-report/skyflow-node.api.md +++ b/skyvault/api-report/skyflow-node.api.md @@ -4,23 +4,69 @@ ```ts +import { ApiKeyCredentials } from 'skyflow-common'; +import { AuthInfo } from 'skyflow-common'; +import { BaseDeleteRequest } from 'skyflow-common'; +import { BaseDeleteResponse } from 'skyflow-common'; +import { BaseDetokenizeRequest } from 'skyflow-common'; +import { BaseDetokenizeResponse } from 'skyflow-common'; +import { BaseFileUploadRequest } from 'skyflow-common'; +import { BaseFileUploadResponse } from 'skyflow-common'; +import { BaseGetRequest } from 'skyflow-common'; +import { BaseGetResponse } from 'skyflow-common'; +import { BaseInsertRequest } from 'skyflow-common'; +import { BaseInsertResponse } from 'skyflow-common'; +import { BaseInvokeConnectionRequest } from 'skyflow-common'; +import { BaseInvokeConnectionResponse } from 'skyflow-common'; +import { BaseOptions } from 'skyflow-common'; +import { BaseQueryRequest } from 'skyflow-common'; +import { BaseQueryResponse } from 'skyflow-common'; +import { BaseSkyflow } from 'skyflow-common'; +import { BaseTokenizeRequest } from 'skyflow-common'; +import { BaseTokenizeResponse } from 'skyflow-common'; +import { BaseUpdateRequest } from 'skyflow-common'; +import { BaseUpdateResponse } from 'skyflow-common'; +import { BaseVaultEntry } from 'skyflow-common'; +import { BearerTokenOptions } from 'skyflow-common'; import { Blob as Blob_2 } from 'buffer'; +import { ConnectionConfig } from 'skyflow-common'; +import { Credentials } from 'skyflow-common'; +import { DetectEntities } from 'skyflow-common'; +import { DetectOutputTranscription } from 'skyflow-common'; +import { Env } from 'skyflow-common'; +import { FileInput } from 'skyflow-common'; import * as fs from 'fs'; - -// @public (undocumented) -export interface ApiKeyCredentials { - // (undocumented) - apiKey: string; -} - -// @public (undocumented) -export type BearerTokenOptions = { - ctx?: string | Record; - roleIDs?: string[]; - roleIds?: string[]; - logLevel?: LogLevel; - tokenUri?: string; -}; +import { generateBearerToken } from 'skyflow-common'; +import { generateBearerTokenFromCreds } from 'skyflow-common'; +import { generateSignedDataTokens } from 'skyflow-common'; +import { generateSignedDataTokensFromCreds } from 'skyflow-common'; +import { GenerateTokenOptions } from 'skyflow-common'; +import { GetResponseData } from 'skyflow-common'; +import { IndexRange } from 'skyflow-common'; +import { InsertResponseType } from 'skyflow-common'; +import { isExpired } from 'skyflow-common'; +import { LogLevel } from 'skyflow-common'; +import { MaskingMethod } from 'skyflow-common'; +import { OrderByEnum } from 'skyflow-common'; +import { PathCredentials } from 'skyflow-common'; +import { QueryResponseType } from 'skyflow-common'; +import { RedactionType } from 'skyflow-common'; +import { RequestMethod } from 'skyflow-common'; +import { SignedDataTokensOptions } from 'skyflow-common'; +import { SkyflowConfig } from 'skyflow-common'; +import { SkyflowError } from 'skyflow-common'; +import { SkyflowRecordError } from 'skyflow-common'; +import { StringCredentials } from 'skyflow-common'; +import { StringKeyValueMapType } from 'skyflow-common'; +import { TokenCredentials } from 'skyflow-common'; +import { TokenizeRequestType } from 'skyflow-common'; +import { TokenMode } from 'skyflow-common'; +import { TokenType } from 'skyflow-common'; +import { VaultConfig } from 'skyflow-common'; + +export { ApiKeyCredentials } + +export { BearerTokenOptions } // @public (undocumented) export class Bleep { @@ -42,18 +88,9 @@ export class Bleep { setStopPadding(stopPadding: number): void; } -// @public (undocumented) -export interface ConnectionConfig { - // (undocumented) - connectionId: string; - // (undocumented) - connectionUrl: string; - // (undocumented) - credentials?: Credentials; -} +export { ConnectionConfig } -// @public (undocumented) -export type Credentials = TokenCredentials | PathCredentials | StringCredentials | ApiKeyCredentials; +export { Credentials } // @public (undocumented) export class DeidentifyFileOptions { @@ -251,7 +288,7 @@ export class DeidentifyTextResponse { } // @public (undocumented) -export class DeleteRequest { +export class DeleteRequest extends BaseDeleteRequest { constructor(table: string, deleteIds: Array); // (undocumented) get ids(): Array; @@ -262,7 +299,7 @@ export class DeleteRequest { } // @public (undocumented) -export class DeleteResponse { +export class DeleteResponse extends BaseDeleteResponse { // @deprecated constructor(input: { deletedIds: Array; @@ -274,161 +311,9 @@ export class DeleteResponse { errors: Array | null; } -// @public (undocumented) -export enum DetectEntities { - // (undocumented) - ACCOUNT_NUMBER = "account_number", - // (undocumented) - AGE = "age", - // (undocumented) - ALL = "all", - // (undocumented) - BANK_ACCOUNT = "bank_account", - // (undocumented) - BLOOD_TYPE = "blood_type", - // (undocumented) - CONDITION = "condition", - // (undocumented) - CORPORATE_ACTION = "corporate_action", - // (undocumented) - CREDIT_CARD = "credit_card", - // (undocumented) - CREDIT_CARD_EXPIRATION = "credit_card_expiration", - // (undocumented) - CVV = "cvv", - // (undocumented) - DATE = "date", - // (undocumented) - DATE_INTERVAL = "date_interval", - // (undocumented) - DAY = "day", - // (undocumented) - DOB = "dob", - // (undocumented) - DOSE = "dose", - // (undocumented) - DRIVER_LICENSE = "driver_license", - // (undocumented) - DRUG = "drug", - // (undocumented) - DURATION = "duration", - // (undocumented) - EFFECT = "effect", - // (undocumented) - EMAIL_ADDRESS = "email_address", - // (undocumented) - EVENT = "event", - // (undocumented) - FILENAME = "filename", - // (undocumented) - FINANCIAL_METRIC = "financial_metric", - // (undocumented) - GENDER = "gender", - // (undocumented) - HEALTHCARE_NUMBER = "healthcare_number", - // (undocumented) - INJURY = "injury", - // (undocumented) - IP_ADDRESS = "ip_address", - // (undocumented) - LANGUAGE = "language", - // (undocumented) - LOCATION = "location", - // (undocumented) - LOCATION_ADDRESS = "location_address", - // (undocumented) - LOCATION_ADDRESS_STREET = "location_address_street", - // (undocumented) - LOCATION_CITY = "location_city", - // (undocumented) - LOCATION_COORDINATE = "location_coordinate", - // (undocumented) - LOCATION_COUNTRY = "location_country", - // (undocumented) - LOCATION_STATE = "location_state", - // (undocumented) - LOCATION_ZIP = "location_zip", - // (undocumented) - MARITAL_STATUS = "marital_status", - // (undocumented) - MEDICAL_CODE = "medical_code", - // (undocumented) - MEDICAL_PROCESS = "medical_process", - // (undocumented) - MONEY = "money", - // (undocumented) - MONTH = "month", - // (undocumented) - NAME = "name", - // (undocumented) - NAME_FAMILY = "name_family", - // (undocumented) - NAME_GIVEN = "name_given", - // (undocumented) - NAME_MEDICAL_PROFESSIONAL = "name_medical_professional", - // (undocumented) - NUMERICAL_PII = "numerical_pii", - // (undocumented) - OCCUPATION = "occupation", - // (undocumented) - ORGANIZATION = "organization", - // (undocumented) - ORGANIZATION_ID = "organization_id", - // (undocumented) - ORGANIZATION_MEDICAL_FACILITY = "organization_medical_facility", - // (undocumented) - ORIGIN = "origin", - // (undocumented) - PASSPORT_NUMBER = "passport_number", - // (undocumented) - PASSWORD = "password", - // (undocumented) - PHONE_NUMBER = "phone_number", - // (undocumented) - PHYSICAL_ATTRIBUTE = "physical_attribute", - // (undocumented) - POLITICAL_AFFILIATION = "political_affiliation", - // (undocumented) - PRODUCT = "product", - // (undocumented) - PROJECT = "project", - // (undocumented) - RELIGION = "religion", - // (undocumented) - ROUTING_NUMBER = "routing_number", - // (undocumented) - SEXUALITY = "sexuality", - // (undocumented) - SSN = "ssn", - // (undocumented) - STATISTICS = "statistics", - // (undocumented) - TIME = "time", - // (undocumented) - TREND = "trend", - // (undocumented) - URL = "url", - // (undocumented) - USERNAME = "username", - // (undocumented) - VEHICLE_ID = "vehicle_id", - // (undocumented) - ZODIAC_SIGN = "zodiac_sign" -} +export { DetectEntities } -// @public (undocumented) -export enum DetectOutputTranscription { - // (undocumented) - DIARIZED_TRANSCRIPTION = "diarized_transcription", - // (undocumented) - MEDICAL_DIARIZED_TRANSCRIPTION = "medical_diarized_transcription", - // (undocumented) - MEDICAL_TRANSCRIPTION = "medical_transcription", - // (undocumented) - PLAINTEXT_TRANSCRIPTION = "plaintext_transcription", - // (undocumented) - TRANSCRIPTION = "transcription" -} +export { DetectOutputTranscription } // @public (undocumented) export interface DetokenizeData { @@ -439,7 +324,7 @@ export interface DetokenizeData { } // @public (undocumented) -export class DetokenizeOptions { +export class DetokenizeOptions extends BaseOptions { constructor(); // (undocumented) getContinueOnError(): boolean | undefined; @@ -456,7 +341,7 @@ export class DetokenizeOptions { } // @public (undocumented) -export class DetokenizeRequest { +export class DetokenizeRequest extends BaseDetokenizeRequest { constructor(data: DetokenizeData[]); // (undocumented) get data(): DetokenizeData[]; @@ -464,7 +349,7 @@ export class DetokenizeRequest { } // @public (undocumented) -export class DetokenizeResponse { +export class DetokenizeResponse extends BaseDetokenizeResponse { constructor(input: { detokenizedFields: Array | null; errors: Array | null; @@ -477,26 +362,12 @@ export class DetokenizeResponse { errors: Array | null; } -// @public (undocumented) -export enum Env { - // (undocumented) - DEV = "DEV", - // (undocumented) - PROD = "PROD", - // (undocumented) - SANDBOX = "SANDBOX", - // (undocumented) - STAGE = "STAGE" -} +export { Env } -// Warning: (ae-forgotten-export) The symbol "Filepath" needs to be exported by the entry point index.d.ts -// Warning: (ae-forgotten-export) The symbol "FileObject" needs to be exported by the entry point index.d.ts -// -// @public (undocumented) -export type FileInput = Filepath | FileObject; +export { FileInput } // @public (undocumented) -export class FileUploadOptions { +export class FileUploadOptions extends BaseOptions { constructor(); // (undocumented) getBase64(): string | undefined; @@ -521,13 +392,11 @@ export class FileUploadOptions { } // @public (undocumented) -export class FileUploadRequest { +export class FileUploadRequest extends BaseFileUploadRequest { constructor(table: string, columnNameOrSkyflowId: string, columnName?: string); // (undocumented) get columnName(): string; set columnName(value: string); - // @internal (undocumented) - getLegacySkyflowId(): string | undefined; // @deprecated (undocumented) get skyflowId(): string; set skyflowId(value: string); @@ -537,7 +406,7 @@ export class FileUploadRequest { } // @public (undocumented) -export class FileUploadResponse { +export class FileUploadResponse extends BaseFileUploadResponse { constructor(input: { skyflowId: string; errors: Array | null; @@ -548,26 +417,15 @@ export class FileUploadResponse { skyflowId: string; } -// Warning: (ae-forgotten-export) The symbol "TokenResponse" needs to be exported by the entry point index.d.ts -// -// @public (undocumented) -export function generateBearerToken(credentialsFilePath: string, options?: BearerTokenOptions): Promise; +export { generateBearerToken } -// @public (undocumented) -export function generateBearerTokenFromCreds(credentials: any, options?: BearerTokenOptions): Promise; +export { generateBearerTokenFromCreds } -// Warning: (ae-forgotten-export) The symbol "SignedDataTokensResponse" needs to be exported by the entry point index.d.ts -// -// @public (undocumented) -export function generateSignedDataTokens(credentialsFilePath: string, options: SignedDataTokensOptions): Promise; +export { generateSignedDataTokens } -// @public (undocumented) -export function generateSignedDataTokensFromCreds(credentials: any, options: SignedDataTokensOptions): Promise; +export { generateSignedDataTokensFromCreds } -// @public (undocumented) -export type GenerateTokenOptions = { - logLevel?: LogLevel; -}; +export { GenerateTokenOptions } // @public (undocumented) export class GetColumnRequest { @@ -592,7 +450,7 @@ export class GetDetectRunRequest { } // @public (undocumented) -export class GetOptions { +export class GetOptions extends BaseOptions { constructor(); // (undocumented) getColumnName(): string | undefined; @@ -637,7 +495,7 @@ export class GetOptions { } // @public (undocumented) -export class GetRequest { +export class GetRequest extends BaseGetRequest { constructor(table: string, _ids: Array); // (undocumented) get ids(): Array; @@ -648,7 +506,7 @@ export class GetRequest { } // @public (undocumented) -export class GetResponse { +export class GetResponse extends BaseGetResponse { constructor(input: { data: Array; errors: Array | null; @@ -659,22 +517,12 @@ export class GetResponse { errors: Array | null; } -// @public (undocumented) -export interface GetResponseData { - // (undocumented) - [key: string]: unknown; -} +export { GetResponseData } -// @public (undocumented) -export interface IndexRange { - // (undocumented) - end?: number; - // (undocumented) - start?: number; -} +export { IndexRange } // @public (undocumented) -export class InsertOptions { +export class InsertOptions extends BaseOptions { constructor(); // (undocumented) getContinueOnError(): boolean | undefined; @@ -703,7 +551,7 @@ export class InsertOptions { } // @public (undocumented) -export class InsertRequest { +export class InsertRequest extends BaseInsertRequest { constructor(table: string, data: Record[]); // (undocumented) get data(): Record[]; @@ -714,7 +562,7 @@ export class InsertRequest { } // @public (undocumented) -export class InsertResponse { +export class InsertResponse extends BaseInsertResponse { // @deprecated constructor(input: { insertedFields: Array; @@ -726,18 +574,10 @@ export class InsertResponse { insertedFields: Array; } -// @public (undocumented) -export interface InsertResponseType { - // (undocumented) - [key: string]: unknown; - // @deprecated (undocumented) - skyflow_id?: string; - // (undocumented) - skyflowId: string; -} +export { InsertResponseType } // @public (undocumented) -export class InvokeConnectionRequest { +export class InvokeConnectionRequest extends BaseInvokeConnectionRequest { constructor(method: RequestMethod, body?: StringKeyValueMapType, headers?: StringKeyValueMapType, pathParams?: StringKeyValueMapType, queryParams?: StringKeyValueMapType); // (undocumented) body?: StringKeyValueMapType; @@ -752,7 +592,7 @@ export class InvokeConnectionRequest { } // @public (undocumented) -export class InvokeConnectionResponse { +export class InvokeConnectionResponse extends BaseInvokeConnectionResponse { constructor(input: { data?: object; metadata?: Record; @@ -766,55 +606,18 @@ export class InvokeConnectionResponse { metadata?: Record; } -// @public (undocumented) -export function isExpired(token: string): boolean; +export { isExpired } -// @public (undocumented) -export enum LogLevel { - // (undocumented) - DEBUG = "DEBUG", - // (undocumented) - ERROR = "ERROR", - // (undocumented) - INFO = "INFO", - // (undocumented) - OFF = "OFF", - // (undocumented) - WARN = "WARN" -} +export { LogLevel } -// @public (undocumented) -export enum MaskingMethod { - // (undocumented) - Blackbox = "blackbox", - // (undocumented) - Blur = "blur" -} +export { MaskingMethod } -// @public (undocumented) -export enum OrderByEnum { - // (undocumented) - ASCENDING = "ASCENDING", - // (undocumented) - DESCENDING = "DESCENDING", - // (undocumented) - NONE = "NONE" -} +export { OrderByEnum } -// @public (undocumented) -export interface PathCredentials { - // (undocumented) - context?: string | Record; - // (undocumented) - path: string; - // (undocumented) - roles?: Array; - // (undocumented) - tokenUri?: string; -} +export { PathCredentials } // @public (undocumented) -export class QueryRequest { +export class QueryRequest extends BaseQueryRequest { constructor(query: string); // (undocumented) get query(): string; @@ -822,7 +625,7 @@ export class QueryRequest { } // @public (undocumented) -export class QueryResponse { +export class QueryResponse extends BaseQueryResponse { constructor(input: { fields: Array; errors: Array | null; @@ -833,23 +636,9 @@ export class QueryResponse { fields: Array; } -// @public (undocumented) -export interface QueryResponseType { - // (undocumented) - [key: string]: unknown; -} +export { QueryResponseType } -// @public (undocumented) -export enum RedactionType { - // (undocumented) - DEFAULT = "DEFAULT", - // (undocumented) - MASKED = "MASKED", - // (undocumented) - PLAIN_TEXT = "PLAIN_TEXT", - // (undocumented) - REDACTED = "REDACTED" -} +export { RedactionType } // @public (undocumented) export class ReidentifyTextOptions { @@ -884,34 +673,23 @@ export class ReidentifyTextResponse { processedText: string; } -// @public (undocumented) -export enum RequestMethod { - // (undocumented) - GET = "GET", - // (undocumented) - PATCH = "PATCH", - // (undocumented) - POST = "POST", - // (undocumented) - PUT = "PUT" -} +export { RequestMethod } -// @public (undocumented) -export type SignedDataTokensOptions = { - dataTokens: string[]; - timeToLive?: number; - ctx?: string | Record; - logLevel?: LogLevel; - tokenUri?: string; -}; +export { SignedDataTokensOptions } -// @public (undocumented) -export class Skyflow { +// Warning: (ae-forgotten-export) The symbol "VaultEntry" needs to be exported by the entry point index.d.ts +// +// @public +export class Skyflow extends BaseSkyflow { constructor(config: SkyflowConfig); // (undocumented) addConnectionConfig(config: ConnectionConfig): void; // (undocumented) - addVaultConfig(config: VaultConfig): void; + protected applyCredentials(): void; + // (undocumented) + protected applyLogLevel(): void; + // (undocumented) + protected buildVaultEntry(config: VaultConfig): VaultEntry; // Warning: (ae-forgotten-export) The symbol "ConnectionController" needs to be exported by the entry point index.d.ts // // (undocumented) @@ -921,93 +699,28 @@ export class Skyflow { // (undocumented) detect(vaultId?: string): DetectController; // (undocumented) - getConnectionConfig(connectionId: string): ConnectionConfig | VaultConfig; - // (undocumented) - getLogLevel(): LogLevel; - // (undocumented) - getSkyflowCredentials(): Credentials | undefined; - // (undocumented) - getVaultConfig(vaultId: string): ConnectionConfig | VaultConfig; + getConnectionConfig(connectionId: string): ConnectionConfig; // (undocumented) removeConnectionConfig(connectionId: string): void; // (undocumented) - removeVaultConfig(vaultId: string): void; - // (undocumented) - setLogLevel(logLevel: LogLevel): void; - // (undocumented) updateConnectionConfig(config: ConnectionConfig): void; - // (undocumented) - updateLogLevel(logLevel: LogLevel): Skyflow; - // (undocumented) - updateSkyflowCredentials(credentials: Credentials): void; - // (undocumented) - updateVaultConfig(config: VaultConfig): void; // Warning: (ae-forgotten-export) The symbol "VaultController" needs to be exported by the entry point index.d.ts // // (undocumented) vault(vaultId?: string): VaultController; } -// @public (undocumented) -export interface SkyflowConfig { - // (undocumented) - connectionConfigs?: ConnectionConfig[]; - // (undocumented) - logLevel?: LogLevel; - // (undocumented) - skyflowCredentials?: Credentials; - // (undocumented) - vaultConfigs?: VaultConfig[]; -} +export { SkyflowConfig } -// @public (undocumented) -export class SkyflowError extends Error { - constructor(errorCode: ISkyflowError, args?: Array); - // Warning: (ae-forgotten-export) The symbol "ISkyflowError" needs to be exported by the entry point index.d.ts - // - // (undocumented) - error?: ISkyflowError; -} +export { SkyflowError } -// @public (undocumented) -export interface SkyflowRecordError { - // (undocumented) - error: string; - // (undocumented) - httpCode?: string | number | null; - // @deprecated (undocumented) - request_ID?: string | null; - // (undocumented) - requestId: string | null; - // (undocumented) - requestIndex?: number | null; - // (undocumented) - token?: string | null; -} +export { SkyflowRecordError } -// @public (undocumented) -export interface StringCredentials { - // (undocumented) - context?: string | Record; - // (undocumented) - credentialsString: string; - // (undocumented) - roles?: Array; - // (undocumented) - tokenUri?: string; -} +export { StringCredentials } -// @public (undocumented) -export interface StringKeyValueMapType { - // (undocumented) - [key: string]: object | string; -} +export { StringKeyValueMapType } -// @public (undocumented) -export interface TokenCredentials { - // (undocumented) - token: string; -} +export { TokenCredentials } // @public (undocumented) export class TokenFormat { @@ -1030,23 +743,17 @@ export class TokenFormat { } // @public (undocumented) -export class TokenizeRequest { +export class TokenizeRequest extends BaseTokenizeRequest { constructor(values: Array); // (undocumented) get values(): Array; set values(value: Array); } -// @public (undocumented) -export interface TokenizeRequestType { - // (undocumented) - columnGroup: string; - // (undocumented) - value: string; -} +export { TokenizeRequestType } // @public (undocumented) -export class TokenizeResponse { +export class TokenizeResponse extends BaseTokenizeResponse { constructor(input: { tokens: Array; errors: Array | null; @@ -1057,25 +764,9 @@ export class TokenizeResponse { tokens: Array; } -// @public (undocumented) -export enum TokenMode { - // (undocumented) - DISABLE = "DISABLE", - // (undocumented) - ENABLE = "ENABLE", - // (undocumented) - ENABLE_STRICT = "ENABLE_STRICT" -} +export { TokenMode } -// @public (undocumented) -export enum TokenType { - // (undocumented) - ENTITY_ONLY = "entity_only", - // (undocumented) - ENTITY_UNIQUE_COUNTER = "entity_unq_counter", - // (undocumented) - VAULT_TOKEN = "vault_token" -} +export { TokenType } // @public (undocumented) export class Transformations { @@ -1094,7 +785,7 @@ export class Transformations { } // @public (undocumented) -export class UpdateOptions { +export class UpdateOptions extends BaseOptions { constructor(); // (undocumented) getReturnTokens(): boolean | undefined; @@ -1111,7 +802,7 @@ export class UpdateOptions { } // @public (undocumented) -export class UpdateRequest { +export class UpdateRequest extends BaseUpdateRequest { constructor(table: string, data: Record); // (undocumented) get data(): Record; @@ -1122,7 +813,7 @@ export class UpdateRequest { } // @public (undocumented) -export class UpdateResponse { +export class UpdateResponse extends BaseUpdateResponse { constructor(input: { updatedField: InsertResponseType; errors: Array | null; @@ -1133,17 +824,7 @@ export class UpdateResponse { updatedField: InsertResponseType; } -// @public (undocumented) -export interface VaultConfig { - // (undocumented) - clusterId: string; - // (undocumented) - credentials?: Credentials; - // (undocumented) - env?: Env; - // (undocumented) - vaultId: string; -} +export { VaultConfig } // (No @packageDocumentation comment for this package) diff --git a/skyvault/babel.config.js b/skyvault/babel.config.js new file mode 100644 index 00000000..7bcee91b --- /dev/null +++ b/skyvault/babel.config.js @@ -0,0 +1,32 @@ +/* + Copyright (c) 2022 Skyflow, Inc. +*/ +module.exports = function (api) { + api.cache(false); + const presets = [ + ['@babel/preset-typescript'], + [ + '@babel/preset-env', + { + corejs: { version: 3 }, + useBuiltIns: 'usage', + targets: { + edge: '17', + firefox: '60', + chrome: '67', + safari: '11.1', + ie: '11', + }, + }, + ], + ]; + const plugins = [ + ['@babel/plugin-proposal-decorators', { decoratorsBeforeExport: true }], + ['@babel/plugin-transform-class-properties'], + ['@babel/transform-runtime'], + ]; + return { + presets, + plugins, + }; +}; diff --git a/skyvault/jest.config.js b/skyvault/jest.config.js new file mode 100644 index 00000000..4ed9a522 --- /dev/null +++ b/skyvault/jest.config.js @@ -0,0 +1,25 @@ +module.exports = { + transform: { + '^.+\\.(ts|tsx)?$': 'ts-jest', + '^.+\\.(js|jsx)$': 'babel-jest', + }, + // Do NOT babel-transform the already-compiled `common` package (it is plain + // CJS from tsc). Jest may resolve the skyflow-common symlink to its realpath + // (outside node_modules), which would otherwise trigger babel's core-js + // injection against a dep that isn't installed. Matches both the realpath + // (".../common/lib/...") and the symlink path (".../skyflow-common/lib/..."). + transformIgnorePatterns: ['/node_modules/', 'common/lib'], + verbose: true, + collectCoverage: true, + coveragePathIgnorePatterns: [ + "/node_modules/", + "src/ _generated_", + "src/.*/model/request", + "src/.*/model/response", + "src/.*/model/options", + ], + testEnvironmentOptions: { + "url": "https://skyflow-test.com" + }, + testEnvironment: "node", + }; diff --git a/skyvault/package.json b/skyvault/package.json new file mode 100644 index 00000000..ef0681de --- /dev/null +++ b/skyvault/package.json @@ -0,0 +1,60 @@ +{ + "name": "skyflow-node", + "version": "2.1.2", + "description": "Skyflow SDK for Node.js", + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "require": "./dist/index.js", + "default": "./dist/index.js" + } + }, + "files": [ + "dist", + "README.md", + "LICENSE" + ], + "scripts": { + "build": "tsc -b", + "build:dist": "tsup", + "test": "jest --coverage", + "contract-snapshot": "api-extractor run --local", + "contract-snapshot-verify": "api-extractor run" + }, + "repository": { + "type": "git", + "url": "https://github.com/skyflowapi/skyflow-node-sdk.git" + }, + "browser": { + "child_process": false + }, + "author": "", + "license": "MIT", + "keywords": [ + "skyflow", + "privacy", + "api", + "security", + "vault", + "PII", + "tokenization" + ], + "dependencies": { + "dotenv": "16.6.1", + "form-data": "4.0.6", + "form-data-encoder": "4.1.0", + "formdata-node": "6.0.3", + "js-base64": "3.7.7", + "jsonwebtoken": "9.0.3", + "jwt-decode": "4.0.0", + "node-fetch": "2.7.0", + "qs": "6.15.2", + "readable-stream": "4.7.0", + "url-join": "4.0.1" + }, + "devDependencies": { + "skyflow-common": "*" + } +} diff --git a/src/ _generated_/rest/Client.ts b/skyvault/src/ _generated_/rest/Client.ts similarity index 100% rename from src/ _generated_/rest/Client.ts rename to skyvault/src/ _generated_/rest/Client.ts diff --git a/src/ _generated_/rest/api/errors/BadRequestError.ts b/skyvault/src/ _generated_/rest/api/errors/BadRequestError.ts similarity index 100% rename from src/ _generated_/rest/api/errors/BadRequestError.ts rename to skyvault/src/ _generated_/rest/api/errors/BadRequestError.ts diff --git a/skyvault/src/ _generated_/rest/api/errors/InternalServerError.ts b/skyvault/src/ _generated_/rest/api/errors/InternalServerError.ts new file mode 100644 index 00000000..4141147e --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/errors/InternalServerError.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; +import * as core from "../../core"; + +export class InternalServerError extends errors.SkyflowError { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { + super({ + message: "InternalServerError", + statusCode: 500, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, InternalServerError.prototype); + } +} diff --git a/src/ _generated_/rest/api/errors/NotFoundError.ts b/skyvault/src/ _generated_/rest/api/errors/NotFoundError.ts similarity index 100% rename from src/ _generated_/rest/api/errors/NotFoundError.ts rename to skyvault/src/ _generated_/rest/api/errors/NotFoundError.ts diff --git a/src/ _generated_/rest/api/errors/UnauthorizedError.ts b/skyvault/src/ _generated_/rest/api/errors/UnauthorizedError.ts similarity index 100% rename from src/ _generated_/rest/api/errors/UnauthorizedError.ts rename to skyvault/src/ _generated_/rest/api/errors/UnauthorizedError.ts diff --git a/src/ _generated_/rest/api/errors/index.ts b/skyvault/src/ _generated_/rest/api/errors/index.ts similarity index 100% rename from src/ _generated_/rest/api/errors/index.ts rename to skyvault/src/ _generated_/rest/api/errors/index.ts diff --git a/src/ _generated_/rest/api/index.ts b/skyvault/src/ _generated_/rest/api/index.ts similarity index 100% rename from src/ _generated_/rest/api/index.ts rename to skyvault/src/ _generated_/rest/api/index.ts diff --git a/src/ _generated_/rest/api/resources/audit/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/audit/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/client/Client.ts diff --git a/src/ _generated_/rest/api/resources/strings/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/audit/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/client/index.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/client/index.ts diff --git a/src/ _generated_/rest/api/resources/audit/client/requests/AuditServiceListAuditEventsRequest.ts b/skyvault/src/ _generated_/rest/api/resources/audit/client/requests/AuditServiceListAuditEventsRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/client/requests/AuditServiceListAuditEventsRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/client/requests/AuditServiceListAuditEventsRequest.ts diff --git a/src/ _generated_/rest/api/resources/audit/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/audit/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/client/requests/index.ts diff --git a/src/ _generated_/rest/api/resources/files/index.ts b/skyvault/src/ _generated_/rest/api/resources/audit/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/index.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/index.ts diff --git a/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsActionType.ts b/skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsActionType.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsActionType.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsActionType.ts diff --git a/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAccessType.ts b/skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAccessType.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAccessType.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAccessType.ts diff --git a/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextActorType.ts b/skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextActorType.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextActorType.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextActorType.ts diff --git a/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAuthMode.ts b/skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAuthMode.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAuthMode.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsContextAuthMode.ts diff --git a/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsResourceType.ts b/skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsResourceType.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsResourceType.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestFilterOpsResourceType.ts diff --git a/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestSortOpsOrderBy.ts b/skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestSortOpsOrderBy.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestSortOpsOrderBy.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/types/AuditServiceListAuditEventsRequestSortOpsOrderBy.ts diff --git a/src/ _generated_/rest/api/resources/audit/types/index.ts b/skyvault/src/ _generated_/rest/api/resources/audit/types/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/audit/types/index.ts rename to skyvault/src/ _generated_/rest/api/resources/audit/types/index.ts diff --git a/src/ _generated_/rest/api/resources/authentication/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/authentication/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/authentication/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/authentication/client/Client.ts diff --git a/src/ _generated_/rest/api/resources/tokens/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/authentication/client/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/tokens/client/index.ts rename to skyvault/src/ _generated_/rest/api/resources/authentication/client/index.ts diff --git a/src/ _generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts b/skyvault/src/ _generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/authentication/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/authentication/client/requests/index.ts new file mode 100644 index 00000000..635fca22 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/authentication/client/requests/index.ts @@ -0,0 +1 @@ +export { type V1GetAuthTokenRequest } from "./V1GetAuthTokenRequest"; diff --git a/src/ _generated_/rest/api/resources/tokens/index.ts b/skyvault/src/ _generated_/rest/api/resources/authentication/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/tokens/index.ts rename to skyvault/src/ _generated_/rest/api/resources/authentication/index.ts diff --git a/src/ _generated_/rest/api/resources/binLookup/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/binLookup/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/binLookup/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/binLookup/client/Client.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/binLookup/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/binLookup/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/binLookup/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/src/ _generated_/rest/api/resources/binLookup/client/requests/V1BinListRequest.ts b/skyvault/src/ _generated_/rest/api/resources/binLookup/client/requests/V1BinListRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/binLookup/client/requests/V1BinListRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/binLookup/client/requests/V1BinListRequest.ts diff --git a/src/ _generated_/rest/api/resources/binLookup/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/binLookup/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/binLookup/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/binLookup/client/requests/index.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/binLookup/index.ts b/skyvault/src/ _generated_/rest/api/resources/binLookup/index.ts new file mode 100644 index 00000000..5ec76921 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/binLookup/index.ts @@ -0,0 +1 @@ +export * from "./client"; diff --git a/src/ _generated_/rest/api/resources/files/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/Client.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/files/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/files/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyAudioRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyAudioRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyAudioRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyAudioRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyDocumentRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyDocumentRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyDocumentRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyDocumentRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyFileRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyFileRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyFileRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyFileRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyImageRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyImageRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyImageRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyImageRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPdfRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPdfRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPdfRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPdfRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPresentationRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPresentationRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPresentationRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyPresentationRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifySpreadsheetRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifySpreadsheetRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifySpreadsheetRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifySpreadsheetRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyStructuredTextRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyStructuredTextRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyStructuredTextRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyStructuredTextRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyTextRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyTextRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/DeidentifyTextRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/DeidentifyTextRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/GetRunRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/GetRunRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/GetRunRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/GetRunRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/ReidentifyFileRequest.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/ReidentifyFileRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/ReidentifyFileRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/ReidentifyFileRequest.ts diff --git a/src/ _generated_/rest/api/resources/files/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/files/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/files/client/requests/index.ts diff --git a/src/ _generated_/rest/api/resources/records/index.ts b/skyvault/src/ _generated_/rest/api/resources/files/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/index.ts rename to skyvault/src/ _generated_/rest/api/resources/files/index.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestOutputTranscription.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestOutputTranscription.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestOutputTranscription.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyAudioRequestOutputTranscription.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyDocumentRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyFileRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestMaskingMethod.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestMaskingMethod.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestMaskingMethod.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyImageRequestMaskingMethod.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyPdfRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyPdfRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyPdfRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyPdfRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyPresentationRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifySpreadsheetRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyStructuredTextRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/DeidentifyTextRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyTextRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/DeidentifyTextRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/DeidentifyTextRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFile.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFile.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFile.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFile.ts diff --git a/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFileDataFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFileDataFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFileDataFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFileDataFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFormat.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/ReidentifyFileRequestFormat.ts diff --git a/src/ _generated_/rest/api/resources/files/types/index.ts b/skyvault/src/ _generated_/rest/api/resources/files/types/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/files/types/index.ts rename to skyvault/src/ _generated_/rest/api/resources/files/types/index.ts diff --git a/src/ _generated_/rest/api/resources/guardrails/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/guardrails/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/guardrails/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/guardrails/client/Client.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/guardrails/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/guardrails/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/guardrails/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/src/ _generated_/rest/api/resources/guardrails/client/requests/CheckGuardrailsRequest.ts b/skyvault/src/ _generated_/rest/api/resources/guardrails/client/requests/CheckGuardrailsRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/guardrails/client/requests/CheckGuardrailsRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/guardrails/client/requests/CheckGuardrailsRequest.ts diff --git a/src/ _generated_/rest/api/resources/guardrails/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/guardrails/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/guardrails/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/guardrails/client/requests/index.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/guardrails/index.ts b/skyvault/src/ _generated_/rest/api/resources/guardrails/index.ts new file mode 100644 index 00000000..5ec76921 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/guardrails/index.ts @@ -0,0 +1 @@ +export * from "./client"; diff --git a/src/ _generated_/rest/api/resources/index.ts b/skyvault/src/ _generated_/rest/api/resources/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/index.ts rename to skyvault/src/ _generated_/rest/api/resources/index.ts diff --git a/src/ _generated_/rest/api/resources/query/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/query/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/query/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/query/client/Client.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/query/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/query/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/query/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/src/ _generated_/rest/api/resources/query/client/requests/QueryServiceExecuteQueryBody.ts b/skyvault/src/ _generated_/rest/api/resources/query/client/requests/QueryServiceExecuteQueryBody.ts similarity index 100% rename from src/ _generated_/rest/api/resources/query/client/requests/QueryServiceExecuteQueryBody.ts rename to skyvault/src/ _generated_/rest/api/resources/query/client/requests/QueryServiceExecuteQueryBody.ts diff --git a/src/ _generated_/rest/api/resources/query/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/query/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/query/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/query/client/requests/index.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/query/index.ts b/skyvault/src/ _generated_/rest/api/resources/query/index.ts new file mode 100644 index 00000000..5ec76921 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/query/index.ts @@ -0,0 +1 @@ +export * from "./client"; diff --git a/src/ _generated_/rest/api/resources/records/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/Client.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/records/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/records/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/src/ _generated_/rest/api/resources/records/client/requests/FileServiceUploadFileRequest.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/FileServiceUploadFileRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/FileServiceUploadFileRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/FileServiceUploadFileRequest.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBatchOperationBody.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBatchOperationBody.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBatchOperationBody.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBatchOperationBody.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkDeleteRecordBody.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkDeleteRecordBody.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkDeleteRecordBody.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkDeleteRecordBody.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkGetRecordRequest.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkGetRecordRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkGetRecordRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceBulkGetRecordRequest.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceGetRecordRequest.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceGetRecordRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/RecordServiceGetRecordRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceGetRecordRequest.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceInsertRecordBody.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceInsertRecordBody.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/RecordServiceInsertRecordBody.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceInsertRecordBody.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceUpdateRecordBody.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceUpdateRecordBody.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/RecordServiceUpdateRecordBody.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/RecordServiceUpdateRecordBody.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/UploadFileV2Request.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/UploadFileV2Request.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/UploadFileV2Request.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/UploadFileV2Request.ts diff --git a/src/ _generated_/rest/api/resources/records/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/records/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/records/client/requests/index.ts diff --git a/src/ _generated_/rest/api/resources/strings/index.ts b/skyvault/src/ _generated_/rest/api/resources/records/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/index.ts rename to skyvault/src/ _generated_/rest/api/resources/records/index.ts diff --git a/src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestOrderBy.ts b/skyvault/src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestOrderBy.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestOrderBy.ts rename to skyvault/src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestOrderBy.ts diff --git a/src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestRedaction.ts b/skyvault/src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestRedaction.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestRedaction.ts rename to skyvault/src/ _generated_/rest/api/resources/records/types/RecordServiceBulkGetRecordRequestRedaction.ts diff --git a/src/ _generated_/rest/api/resources/records/types/RecordServiceGetRecordRequestRedaction.ts b/skyvault/src/ _generated_/rest/api/resources/records/types/RecordServiceGetRecordRequestRedaction.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/types/RecordServiceGetRecordRequestRedaction.ts rename to skyvault/src/ _generated_/rest/api/resources/records/types/RecordServiceGetRecordRequestRedaction.ts diff --git a/src/ _generated_/rest/api/resources/records/types/index.ts b/skyvault/src/ _generated_/rest/api/resources/records/types/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/records/types/index.ts rename to skyvault/src/ _generated_/rest/api/resources/records/types/index.ts diff --git a/src/ _generated_/rest/api/resources/strings/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/strings/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/strings/client/Client.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/strings/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/strings/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/strings/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/src/ _generated_/rest/api/resources/strings/client/requests/DeidentifyStringRequest.ts b/skyvault/src/ _generated_/rest/api/resources/strings/client/requests/DeidentifyStringRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/client/requests/DeidentifyStringRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/strings/client/requests/DeidentifyStringRequest.ts diff --git a/src/ _generated_/rest/api/resources/strings/client/requests/ReidentifyStringRequest.ts b/skyvault/src/ _generated_/rest/api/resources/strings/client/requests/ReidentifyStringRequest.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/client/requests/ReidentifyStringRequest.ts rename to skyvault/src/ _generated_/rest/api/resources/strings/client/requests/ReidentifyStringRequest.ts diff --git a/src/ _generated_/rest/api/resources/strings/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/strings/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/strings/client/requests/index.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/strings/index.ts b/skyvault/src/ _generated_/rest/api/resources/strings/index.ts new file mode 100644 index 00000000..c9240f83 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/strings/index.ts @@ -0,0 +1,2 @@ +export * from "./types"; +export * from "./client"; diff --git a/src/ _generated_/rest/api/resources/strings/types/ReidentifyStringRequestFormat.ts b/skyvault/src/ _generated_/rest/api/resources/strings/types/ReidentifyStringRequestFormat.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/types/ReidentifyStringRequestFormat.ts rename to skyvault/src/ _generated_/rest/api/resources/strings/types/ReidentifyStringRequestFormat.ts diff --git a/src/ _generated_/rest/api/resources/strings/types/index.ts b/skyvault/src/ _generated_/rest/api/resources/strings/types/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/strings/types/index.ts rename to skyvault/src/ _generated_/rest/api/resources/strings/types/index.ts diff --git a/src/ _generated_/rest/api/resources/tokens/client/Client.ts b/skyvault/src/ _generated_/rest/api/resources/tokens/client/Client.ts similarity index 100% rename from src/ _generated_/rest/api/resources/tokens/client/Client.ts rename to skyvault/src/ _generated_/rest/api/resources/tokens/client/Client.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/tokens/client/index.ts b/skyvault/src/ _generated_/rest/api/resources/tokens/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/tokens/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/src/ _generated_/rest/api/resources/tokens/client/requests/V1DetokenizePayload.ts b/skyvault/src/ _generated_/rest/api/resources/tokens/client/requests/V1DetokenizePayload.ts similarity index 100% rename from src/ _generated_/rest/api/resources/tokens/client/requests/V1DetokenizePayload.ts rename to skyvault/src/ _generated_/rest/api/resources/tokens/client/requests/V1DetokenizePayload.ts diff --git a/src/ _generated_/rest/api/resources/tokens/client/requests/V1TokenizePayload.ts b/skyvault/src/ _generated_/rest/api/resources/tokens/client/requests/V1TokenizePayload.ts similarity index 100% rename from src/ _generated_/rest/api/resources/tokens/client/requests/V1TokenizePayload.ts rename to skyvault/src/ _generated_/rest/api/resources/tokens/client/requests/V1TokenizePayload.ts diff --git a/src/ _generated_/rest/api/resources/tokens/client/requests/index.ts b/skyvault/src/ _generated_/rest/api/resources/tokens/client/requests/index.ts similarity index 100% rename from src/ _generated_/rest/api/resources/tokens/client/requests/index.ts rename to skyvault/src/ _generated_/rest/api/resources/tokens/client/requests/index.ts diff --git a/skyvault/src/ _generated_/rest/api/resources/tokens/index.ts b/skyvault/src/ _generated_/rest/api/resources/tokens/index.ts new file mode 100644 index 00000000..5ec76921 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/resources/tokens/index.ts @@ -0,0 +1 @@ +export * from "./client"; diff --git a/src/ _generated_/rest/api/types/AllowRegex.ts b/skyvault/src/ _generated_/rest/api/types/AllowRegex.ts similarity index 100% rename from src/ _generated_/rest/api/types/AllowRegex.ts rename to skyvault/src/ _generated_/rest/api/types/AllowRegex.ts diff --git a/src/ _generated_/rest/api/types/AuditEventAuditResourceType.ts b/skyvault/src/ _generated_/rest/api/types/AuditEventAuditResourceType.ts similarity index 100% rename from src/ _generated_/rest/api/types/AuditEventAuditResourceType.ts rename to skyvault/src/ _generated_/rest/api/types/AuditEventAuditResourceType.ts diff --git a/src/ _generated_/rest/api/types/AuditEventContext.ts b/skyvault/src/ _generated_/rest/api/types/AuditEventContext.ts similarity index 100% rename from src/ _generated_/rest/api/types/AuditEventContext.ts rename to skyvault/src/ _generated_/rest/api/types/AuditEventContext.ts diff --git a/src/ _generated_/rest/api/types/AuditEventData.ts b/skyvault/src/ _generated_/rest/api/types/AuditEventData.ts similarity index 100% rename from src/ _generated_/rest/api/types/AuditEventData.ts rename to skyvault/src/ _generated_/rest/api/types/AuditEventData.ts diff --git a/src/ _generated_/rest/api/types/AuditEventHttpInfo.ts b/skyvault/src/ _generated_/rest/api/types/AuditEventHttpInfo.ts similarity index 100% rename from src/ _generated_/rest/api/types/AuditEventHttpInfo.ts rename to skyvault/src/ _generated_/rest/api/types/AuditEventHttpInfo.ts diff --git a/src/ _generated_/rest/api/types/BatchRecordMethod.ts b/skyvault/src/ _generated_/rest/api/types/BatchRecordMethod.ts similarity index 100% rename from src/ _generated_/rest/api/types/BatchRecordMethod.ts rename to skyvault/src/ _generated_/rest/api/types/BatchRecordMethod.ts diff --git a/src/ _generated_/rest/api/types/CheckGuardrailsResponse.ts b/skyvault/src/ _generated_/rest/api/types/CheckGuardrailsResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/CheckGuardrailsResponse.ts rename to skyvault/src/ _generated_/rest/api/types/CheckGuardrailsResponse.ts diff --git a/src/ _generated_/rest/api/types/CheckGuardrailsResponseValidation.ts b/skyvault/src/ _generated_/rest/api/types/CheckGuardrailsResponseValidation.ts similarity index 100% rename from src/ _generated_/rest/api/types/CheckGuardrailsResponseValidation.ts rename to skyvault/src/ _generated_/rest/api/types/CheckGuardrailsResponseValidation.ts diff --git a/src/ _generated_/rest/api/types/ConfigurationId.ts b/skyvault/src/ _generated_/rest/api/types/ConfigurationId.ts similarity index 100% rename from src/ _generated_/rest/api/types/ConfigurationId.ts rename to skyvault/src/ _generated_/rest/api/types/ConfigurationId.ts diff --git a/src/ _generated_/rest/api/types/ContextAccessType.ts b/skyvault/src/ _generated_/rest/api/types/ContextAccessType.ts similarity index 100% rename from src/ _generated_/rest/api/types/ContextAccessType.ts rename to skyvault/src/ _generated_/rest/api/types/ContextAccessType.ts diff --git a/src/ _generated_/rest/api/types/ContextAuthMode.ts b/skyvault/src/ _generated_/rest/api/types/ContextAuthMode.ts similarity index 100% rename from src/ _generated_/rest/api/types/ContextAuthMode.ts rename to skyvault/src/ _generated_/rest/api/types/ContextAuthMode.ts diff --git a/src/ _generated_/rest/api/types/DeidentifyFileOutput.ts b/skyvault/src/ _generated_/rest/api/types/DeidentifyFileOutput.ts similarity index 100% rename from src/ _generated_/rest/api/types/DeidentifyFileOutput.ts rename to skyvault/src/ _generated_/rest/api/types/DeidentifyFileOutput.ts diff --git a/src/ _generated_/rest/api/types/DeidentifyFileOutputProcessedFileType.ts b/skyvault/src/ _generated_/rest/api/types/DeidentifyFileOutputProcessedFileType.ts similarity index 100% rename from src/ _generated_/rest/api/types/DeidentifyFileOutputProcessedFileType.ts rename to skyvault/src/ _generated_/rest/api/types/DeidentifyFileOutputProcessedFileType.ts diff --git a/src/ _generated_/rest/api/types/DeidentifyFileResponse.ts b/skyvault/src/ _generated_/rest/api/types/DeidentifyFileResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/DeidentifyFileResponse.ts rename to skyvault/src/ _generated_/rest/api/types/DeidentifyFileResponse.ts diff --git a/src/ _generated_/rest/api/types/DeidentifyStatusResponse.ts b/skyvault/src/ _generated_/rest/api/types/DeidentifyStatusResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/DeidentifyStatusResponse.ts rename to skyvault/src/ _generated_/rest/api/types/DeidentifyStatusResponse.ts diff --git a/src/ _generated_/rest/api/types/DeidentifyStatusResponseOutputType.ts b/skyvault/src/ _generated_/rest/api/types/DeidentifyStatusResponseOutputType.ts similarity index 100% rename from src/ _generated_/rest/api/types/DeidentifyStatusResponseOutputType.ts rename to skyvault/src/ _generated_/rest/api/types/DeidentifyStatusResponseOutputType.ts diff --git a/src/ _generated_/rest/api/types/DeidentifyStatusResponseStatus.ts b/skyvault/src/ _generated_/rest/api/types/DeidentifyStatusResponseStatus.ts similarity index 100% rename from src/ _generated_/rest/api/types/DeidentifyStatusResponseStatus.ts rename to skyvault/src/ _generated_/rest/api/types/DeidentifyStatusResponseStatus.ts diff --git a/src/ _generated_/rest/api/types/DeidentifyStringResponse.ts b/skyvault/src/ _generated_/rest/api/types/DeidentifyStringResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/DeidentifyStringResponse.ts rename to skyvault/src/ _generated_/rest/api/types/DeidentifyStringResponse.ts diff --git a/src/ _generated_/rest/api/types/DetectedEntity.ts b/skyvault/src/ _generated_/rest/api/types/DetectedEntity.ts similarity index 100% rename from src/ _generated_/rest/api/types/DetectedEntity.ts rename to skyvault/src/ _generated_/rest/api/types/DetectedEntity.ts diff --git a/src/ _generated_/rest/api/types/DetokenizeRecordResponseValueType.ts b/skyvault/src/ _generated_/rest/api/types/DetokenizeRecordResponseValueType.ts similarity index 100% rename from src/ _generated_/rest/api/types/DetokenizeRecordResponseValueType.ts rename to skyvault/src/ _generated_/rest/api/types/DetokenizeRecordResponseValueType.ts diff --git a/src/ _generated_/rest/api/types/EntityLocation.ts b/skyvault/src/ _generated_/rest/api/types/EntityLocation.ts similarity index 100% rename from src/ _generated_/rest/api/types/EntityLocation.ts rename to skyvault/src/ _generated_/rest/api/types/EntityLocation.ts diff --git a/src/ _generated_/rest/api/types/EntityType.ts b/skyvault/src/ _generated_/rest/api/types/EntityType.ts similarity index 100% rename from src/ _generated_/rest/api/types/EntityType.ts rename to skyvault/src/ _generated_/rest/api/types/EntityType.ts diff --git a/src/ _generated_/rest/api/types/EntityTypes.ts b/skyvault/src/ _generated_/rest/api/types/EntityTypes.ts similarity index 100% rename from src/ _generated_/rest/api/types/EntityTypes.ts rename to skyvault/src/ _generated_/rest/api/types/EntityTypes.ts diff --git a/skyvault/src/ _generated_/rest/api/types/ErrorResponse.ts b/skyvault/src/ _generated_/rest/api/types/ErrorResponse.ts new file mode 100644 index 00000000..822dd151 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/types/ErrorResponse.ts @@ -0,0 +1,9 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface ErrorResponse { + error: Skyflow.ErrorResponseError; +} diff --git a/src/ _generated_/rest/api/types/ErrorResponseError.ts b/skyvault/src/ _generated_/rest/api/types/ErrorResponseError.ts similarity index 100% rename from src/ _generated_/rest/api/types/ErrorResponseError.ts rename to skyvault/src/ _generated_/rest/api/types/ErrorResponseError.ts diff --git a/skyvault/src/ _generated_/rest/api/types/GooglerpcStatus.ts b/skyvault/src/ _generated_/rest/api/types/GooglerpcStatus.ts new file mode 100644 index 00000000..70e20691 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/types/GooglerpcStatus.ts @@ -0,0 +1,11 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface GooglerpcStatus { + code?: number; + message?: string; + details?: Skyflow.ProtobufAny[]; +} diff --git a/skyvault/src/ _generated_/rest/api/types/ProtobufAny.ts b/skyvault/src/ _generated_/rest/api/types/ProtobufAny.ts new file mode 100644 index 00000000..9f90a613 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/types/ProtobufAny.ts @@ -0,0 +1,7 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface ProtobufAny { + "@type"?: string; +} diff --git a/src/ _generated_/rest/api/types/RedactionEnumRedaction.ts b/skyvault/src/ _generated_/rest/api/types/RedactionEnumRedaction.ts similarity index 100% rename from src/ _generated_/rest/api/types/RedactionEnumRedaction.ts rename to skyvault/src/ _generated_/rest/api/types/RedactionEnumRedaction.ts diff --git a/src/ _generated_/rest/api/types/ReidentifyFileResponse.ts b/skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/ReidentifyFileResponse.ts rename to skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponse.ts diff --git a/src/ _generated_/rest/api/types/ReidentifyFileResponseOutput.ts b/skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponseOutput.ts similarity index 100% rename from src/ _generated_/rest/api/types/ReidentifyFileResponseOutput.ts rename to skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponseOutput.ts diff --git a/src/ _generated_/rest/api/types/ReidentifyFileResponseOutputType.ts b/skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponseOutputType.ts similarity index 100% rename from src/ _generated_/rest/api/types/ReidentifyFileResponseOutputType.ts rename to skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponseOutputType.ts diff --git a/src/ _generated_/rest/api/types/ReidentifyFileResponseStatus.ts b/skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponseStatus.ts similarity index 100% rename from src/ _generated_/rest/api/types/ReidentifyFileResponseStatus.ts rename to skyvault/src/ _generated_/rest/api/types/ReidentifyFileResponseStatus.ts diff --git a/src/ _generated_/rest/api/types/ReidentifyStringResponse.ts b/skyvault/src/ _generated_/rest/api/types/ReidentifyStringResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/ReidentifyStringResponse.ts rename to skyvault/src/ _generated_/rest/api/types/ReidentifyStringResponse.ts diff --git a/src/ _generated_/rest/api/types/RequestActionType.ts b/skyvault/src/ _generated_/rest/api/types/RequestActionType.ts similarity index 100% rename from src/ _generated_/rest/api/types/RequestActionType.ts rename to skyvault/src/ _generated_/rest/api/types/RequestActionType.ts diff --git a/src/ _generated_/rest/api/types/ResourceId.ts b/skyvault/src/ _generated_/rest/api/types/ResourceId.ts similarity index 100% rename from src/ _generated_/rest/api/types/ResourceId.ts rename to skyvault/src/ _generated_/rest/api/types/ResourceId.ts diff --git a/src/ _generated_/rest/api/types/RestrictRegex.ts b/skyvault/src/ _generated_/rest/api/types/RestrictRegex.ts similarity index 100% rename from src/ _generated_/rest/api/types/RestrictRegex.ts rename to skyvault/src/ _generated_/rest/api/types/RestrictRegex.ts diff --git a/src/ _generated_/rest/api/types/TokenType.ts b/skyvault/src/ _generated_/rest/api/types/TokenType.ts similarity index 100% rename from src/ _generated_/rest/api/types/TokenType.ts rename to skyvault/src/ _generated_/rest/api/types/TokenType.ts diff --git a/src/ _generated_/rest/api/types/TokenTypeDefault.ts b/skyvault/src/ _generated_/rest/api/types/TokenTypeDefault.ts similarity index 100% rename from src/ _generated_/rest/api/types/TokenTypeDefault.ts rename to skyvault/src/ _generated_/rest/api/types/TokenTypeDefault.ts diff --git a/src/ _generated_/rest/api/types/TokenTypeWithoutVault.ts b/skyvault/src/ _generated_/rest/api/types/TokenTypeWithoutVault.ts similarity index 100% rename from src/ _generated_/rest/api/types/TokenTypeWithoutVault.ts rename to skyvault/src/ _generated_/rest/api/types/TokenTypeWithoutVault.ts diff --git a/src/ _generated_/rest/api/types/TokenTypeWithoutVaultDefault.ts b/skyvault/src/ _generated_/rest/api/types/TokenTypeWithoutVaultDefault.ts similarity index 100% rename from src/ _generated_/rest/api/types/TokenTypeWithoutVaultDefault.ts rename to skyvault/src/ _generated_/rest/api/types/TokenTypeWithoutVaultDefault.ts diff --git a/src/ _generated_/rest/api/types/Transformations.ts b/skyvault/src/ _generated_/rest/api/types/Transformations.ts similarity index 100% rename from src/ _generated_/rest/api/types/Transformations.ts rename to skyvault/src/ _generated_/rest/api/types/Transformations.ts diff --git a/src/ _generated_/rest/api/types/TransformationsShiftDates.ts b/skyvault/src/ _generated_/rest/api/types/TransformationsShiftDates.ts similarity index 100% rename from src/ _generated_/rest/api/types/TransformationsShiftDates.ts rename to skyvault/src/ _generated_/rest/api/types/TransformationsShiftDates.ts diff --git a/src/ _generated_/rest/api/types/TransformationsShiftDatesEntityTypesItem.ts b/skyvault/src/ _generated_/rest/api/types/TransformationsShiftDatesEntityTypesItem.ts similarity index 100% rename from src/ _generated_/rest/api/types/TransformationsShiftDatesEntityTypesItem.ts rename to skyvault/src/ _generated_/rest/api/types/TransformationsShiftDatesEntityTypesItem.ts diff --git a/src/ _generated_/rest/api/types/UploadFileV2Response.ts b/skyvault/src/ _generated_/rest/api/types/UploadFileV2Response.ts similarity index 100% rename from src/ _generated_/rest/api/types/UploadFileV2Response.ts rename to skyvault/src/ _generated_/rest/api/types/UploadFileV2Response.ts diff --git a/src/ _generated_/rest/api/types/Uuid.ts b/skyvault/src/ _generated_/rest/api/types/Uuid.ts similarity index 100% rename from src/ _generated_/rest/api/types/Uuid.ts rename to skyvault/src/ _generated_/rest/api/types/Uuid.ts diff --git a/src/ _generated_/rest/api/types/V1AuditAfterOptions.ts b/skyvault/src/ _generated_/rest/api/types/V1AuditAfterOptions.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1AuditAfterOptions.ts rename to skyvault/src/ _generated_/rest/api/types/V1AuditAfterOptions.ts diff --git a/src/ _generated_/rest/api/types/V1AuditEventResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1AuditEventResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1AuditEventResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1AuditEventResponse.ts diff --git a/src/ _generated_/rest/api/types/V1AuditResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1AuditResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1AuditResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1AuditResponse.ts diff --git a/src/ _generated_/rest/api/types/V1AuditResponseEvent.ts b/skyvault/src/ _generated_/rest/api/types/V1AuditResponseEvent.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1AuditResponseEvent.ts rename to skyvault/src/ _generated_/rest/api/types/V1AuditResponseEvent.ts diff --git a/src/ _generated_/rest/api/types/V1AuditResponseEventRequest.ts b/skyvault/src/ _generated_/rest/api/types/V1AuditResponseEventRequest.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1AuditResponseEventRequest.ts rename to skyvault/src/ _generated_/rest/api/types/V1AuditResponseEventRequest.ts diff --git a/src/ _generated_/rest/api/types/V1BatchOperationResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1BatchOperationResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1BatchOperationResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1BatchOperationResponse.ts diff --git a/src/ _generated_/rest/api/types/V1BatchRecord.ts b/skyvault/src/ _generated_/rest/api/types/V1BatchRecord.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1BatchRecord.ts rename to skyvault/src/ _generated_/rest/api/types/V1BatchRecord.ts diff --git a/src/ _generated_/rest/api/types/V1BinListResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1BinListResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1BinListResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1BinListResponse.ts diff --git a/src/ _generated_/rest/api/types/V1BulkDeleteRecordResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1BulkDeleteRecordResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1BulkDeleteRecordResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1BulkDeleteRecordResponse.ts diff --git a/src/ _generated_/rest/api/types/V1BulkGetRecordResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1BulkGetRecordResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1BulkGetRecordResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1BulkGetRecordResponse.ts diff --git a/src/ _generated_/rest/api/types/V1Byot.ts b/skyvault/src/ _generated_/rest/api/types/V1Byot.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1Byot.ts rename to skyvault/src/ _generated_/rest/api/types/V1Byot.ts diff --git a/src/ _generated_/rest/api/types/V1Card.ts b/skyvault/src/ _generated_/rest/api/types/V1Card.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1Card.ts rename to skyvault/src/ _generated_/rest/api/types/V1Card.ts diff --git a/src/ _generated_/rest/api/types/V1DeleteFileResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1DeleteFileResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1DeleteFileResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1DeleteFileResponse.ts diff --git a/src/ _generated_/rest/api/types/V1DeleteRecordResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1DeleteRecordResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1DeleteRecordResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1DeleteRecordResponse.ts diff --git a/src/ _generated_/rest/api/types/V1DetokenizeRecordRequest.ts b/skyvault/src/ _generated_/rest/api/types/V1DetokenizeRecordRequest.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1DetokenizeRecordRequest.ts rename to skyvault/src/ _generated_/rest/api/types/V1DetokenizeRecordRequest.ts diff --git a/src/ _generated_/rest/api/types/V1DetokenizeRecordResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1DetokenizeRecordResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1DetokenizeRecordResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1DetokenizeRecordResponse.ts diff --git a/src/ _generated_/rest/api/types/V1DetokenizeResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1DetokenizeResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1DetokenizeResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1DetokenizeResponse.ts diff --git a/src/ _generated_/rest/api/types/V1FieldRecords.ts b/skyvault/src/ _generated_/rest/api/types/V1FieldRecords.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1FieldRecords.ts rename to skyvault/src/ _generated_/rest/api/types/V1FieldRecords.ts diff --git a/src/ _generated_/rest/api/types/V1FileAvScanStatus.ts b/skyvault/src/ _generated_/rest/api/types/V1FileAvScanStatus.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1FileAvScanStatus.ts rename to skyvault/src/ _generated_/rest/api/types/V1FileAvScanStatus.ts diff --git a/skyvault/src/ _generated_/rest/api/types/V1GetAuthTokenResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1GetAuthTokenResponse.ts new file mode 100644 index 00000000..860942f1 --- /dev/null +++ b/skyvault/src/ _generated_/rest/api/types/V1GetAuthTokenResponse.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface V1GetAuthTokenResponse { + /** AccessToken. */ + accessToken?: string; + /** TokenType : Bearer. */ + tokenType?: string; +} diff --git a/src/ _generated_/rest/api/types/V1GetFileScanStatusResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1GetFileScanStatusResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1GetFileScanStatusResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1GetFileScanStatusResponse.ts diff --git a/src/ _generated_/rest/api/types/V1GetQueryResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1GetQueryResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1GetQueryResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1GetQueryResponse.ts diff --git a/src/ _generated_/rest/api/types/V1InsertRecordResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1InsertRecordResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1InsertRecordResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1InsertRecordResponse.ts diff --git a/src/ _generated_/rest/api/types/V1MemberType.ts b/skyvault/src/ _generated_/rest/api/types/V1MemberType.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1MemberType.ts rename to skyvault/src/ _generated_/rest/api/types/V1MemberType.ts diff --git a/src/ _generated_/rest/api/types/V1RecordMetaProperties.ts b/skyvault/src/ _generated_/rest/api/types/V1RecordMetaProperties.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1RecordMetaProperties.ts rename to skyvault/src/ _generated_/rest/api/types/V1RecordMetaProperties.ts diff --git a/src/ _generated_/rest/api/types/V1TokenizeRecordRequest.ts b/skyvault/src/ _generated_/rest/api/types/V1TokenizeRecordRequest.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1TokenizeRecordRequest.ts rename to skyvault/src/ _generated_/rest/api/types/V1TokenizeRecordRequest.ts diff --git a/src/ _generated_/rest/api/types/V1TokenizeRecordResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1TokenizeRecordResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1TokenizeRecordResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1TokenizeRecordResponse.ts diff --git a/src/ _generated_/rest/api/types/V1TokenizeResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1TokenizeResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1TokenizeResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1TokenizeResponse.ts diff --git a/src/ _generated_/rest/api/types/V1UpdateRecordResponse.ts b/skyvault/src/ _generated_/rest/api/types/V1UpdateRecordResponse.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1UpdateRecordResponse.ts rename to skyvault/src/ _generated_/rest/api/types/V1UpdateRecordResponse.ts diff --git a/src/ _generated_/rest/api/types/V1VaultFieldMapping.ts b/skyvault/src/ _generated_/rest/api/types/V1VaultFieldMapping.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1VaultFieldMapping.ts rename to skyvault/src/ _generated_/rest/api/types/V1VaultFieldMapping.ts diff --git a/src/ _generated_/rest/api/types/V1VaultSchemaConfig.ts b/skyvault/src/ _generated_/rest/api/types/V1VaultSchemaConfig.ts similarity index 100% rename from src/ _generated_/rest/api/types/V1VaultSchemaConfig.ts rename to skyvault/src/ _generated_/rest/api/types/V1VaultSchemaConfig.ts diff --git a/src/ _generated_/rest/api/types/VaultId.ts b/skyvault/src/ _generated_/rest/api/types/VaultId.ts similarity index 100% rename from src/ _generated_/rest/api/types/VaultId.ts rename to skyvault/src/ _generated_/rest/api/types/VaultId.ts diff --git a/src/ _generated_/rest/api/types/index.ts b/skyvault/src/ _generated_/rest/api/types/index.ts similarity index 100% rename from src/ _generated_/rest/api/types/index.ts rename to skyvault/src/ _generated_/rest/api/types/index.ts diff --git a/skyvault/src/ _generated_/rest/core/auth/BasicAuth.ts b/skyvault/src/ _generated_/rest/core/auth/BasicAuth.ts new file mode 100644 index 00000000..146df215 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/auth/BasicAuth.ts @@ -0,0 +1,31 @@ +import { Base64 } from "js-base64"; + +export interface BasicAuth { + username: string; + password: string; +} + +const BASIC_AUTH_HEADER_PREFIX = /^Basic /i; + +export const BasicAuth = { + toAuthorizationHeader: (basicAuth: BasicAuth | undefined): string | undefined => { + if (basicAuth == null) { + return undefined; + } + const token = Base64.encode(`${basicAuth.username}:${basicAuth.password}`); + return `Basic ${token}`; + }, + fromAuthorizationHeader: (header: string): BasicAuth => { + const credentials = header.replace(BASIC_AUTH_HEADER_PREFIX, ""); + const decoded = Base64.decode(credentials); + const [username, password] = decoded.split(":", 2); + + if (username == null || password == null) { + throw new Error("Invalid basic auth"); + } + return { + username, + password, + }; + }, +}; diff --git a/skyvault/src/ _generated_/rest/core/auth/BearerToken.ts b/skyvault/src/ _generated_/rest/core/auth/BearerToken.ts new file mode 100644 index 00000000..fe987fc9 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/auth/BearerToken.ts @@ -0,0 +1,15 @@ +export type BearerToken = string; + +const BEARER_AUTH_HEADER_PREFIX = /^Bearer /i; + +export const BearerToken = { + toAuthorizationHeader: (token: BearerToken | undefined): string | undefined => { + if (token == null) { + return undefined; + } + return `Bearer ${token}`; + }, + fromAuthorizationHeader: (header: string): BearerToken => { + return header.replace(BEARER_AUTH_HEADER_PREFIX, "").trim() as BearerToken; + }, +}; diff --git a/skyvault/src/ _generated_/rest/core/auth/index.ts b/skyvault/src/ _generated_/rest/core/auth/index.ts new file mode 100644 index 00000000..ee293b34 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/auth/index.ts @@ -0,0 +1,2 @@ +export { BasicAuth } from "./BasicAuth"; +export { BearerToken } from "./BearerToken"; diff --git a/skyvault/src/ _generated_/rest/core/fetcher/APIResponse.ts b/skyvault/src/ _generated_/rest/core/fetcher/APIResponse.ts new file mode 100644 index 00000000..6335291b --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/APIResponse.ts @@ -0,0 +1,23 @@ +import { RawResponse } from "./RawResponse"; + +/** + * The response of an API call. + * It is a successful response or a failed response. + */ +export type APIResponse = SuccessfulResponse | FailedResponse; + +export interface SuccessfulResponse { + ok: true; + body: T; + /** + * @deprecated Use `rawResponse` instead + */ + headers?: Record; + rawResponse: RawResponse; +} + +export interface FailedResponse { + ok: false; + error: T; + rawResponse: RawResponse; +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/Fetcher.ts b/skyvault/src/ _generated_/rest/core/fetcher/Fetcher.ts new file mode 100644 index 00000000..336ee10b --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/Fetcher.ts @@ -0,0 +1,151 @@ +import { toJson } from "../json"; +import { APIResponse } from "./APIResponse"; +import { abortRawResponse, toRawResponse, unknownRawResponse } from "./RawResponse"; +import { createRequestUrl } from "./createRequestUrl"; +import { getFetchFn } from "./getFetchFn"; +import { getRequestBody } from "./getRequestBody"; +import { getResponseBody } from "./getResponseBody"; +import { makeRequest } from "./makeRequest"; +import { requestWithRetries } from "./requestWithRetries"; + +export type FetchFunction = (args: Fetcher.Args) => Promise>; + +export declare namespace Fetcher { + export interface Args { + url: string; + method: string; + contentType?: string; + headers?: Record; + queryParameters?: Record; + body?: unknown; + timeoutMs?: number; + maxRetries?: number; + withCredentials?: boolean; + abortSignal?: AbortSignal; + requestType?: "json" | "file" | "bytes"; + responseType?: "json" | "blob" | "sse" | "streaming" | "text" | "arrayBuffer"; + duplex?: "half"; + } + + export type Error = FailedStatusCodeError | NonJsonError | TimeoutError | UnknownError; + + export interface FailedStatusCodeError { + reason: "status-code"; + statusCode: number; + body: unknown; + } + + export interface NonJsonError { + reason: "non-json"; + statusCode: number; + rawBody: string; + } + + export interface TimeoutError { + reason: "timeout"; + } + + export interface UnknownError { + reason: "unknown"; + errorMessage: string; + } +} + +export async function fetcherImpl(args: Fetcher.Args): Promise> { + const headers: Record = {}; + if (args.body !== undefined && args.contentType != null) { + headers["Content-Type"] = args.contentType; + } + + if (args.headers != null) { + for (const [key, value] of Object.entries(args.headers)) { + if (value != null) { + headers[key] = value; + } + } + } + + const url = createRequestUrl(args.url, args.queryParameters); + const requestBody: BodyInit | undefined = await getRequestBody({ + body: args.body, + type: args.requestType === "json" ? "json" : "other", + }); + const fetchFn = await getFetchFn(); + + try { + const response = await requestWithRetries( + async () => + makeRequest( + fetchFn, + url, + args.method, + headers, + requestBody, + args.timeoutMs, + args.abortSignal, + args.withCredentials, + args.duplex, + ), + args.maxRetries, + ); + const responseBody = await getResponseBody(response, args.responseType); + + if (response.status >= 200 && response.status < 400) { + return { + ok: true, + body: responseBody as R, + headers: response.headers, + rawResponse: toRawResponse(response), + }; + } else { + return { + ok: false, + error: { + reason: "status-code", + statusCode: response.status, + body: responseBody, + }, + rawResponse: toRawResponse(response), + }; + } + } catch (error) { + if (args.abortSignal != null && args.abortSignal.aborted) { + return { + ok: false, + error: { + reason: "unknown", + errorMessage: "The user aborted a request", + }, + rawResponse: abortRawResponse, + }; + } else if (error instanceof Error && error.name === "AbortError") { + return { + ok: false, + error: { + reason: "timeout", + }, + rawResponse: abortRawResponse, + }; + } else if (error instanceof Error) { + return { + ok: false, + error: { + reason: "unknown", + errorMessage: error.message, + }, + rawResponse: unknownRawResponse, + }; + } + + return { + ok: false, + error: { + reason: "unknown", + errorMessage: toJson(error), + }, + rawResponse: unknownRawResponse, + }; + } +} + +export const fetcher: FetchFunction = fetcherImpl; diff --git a/skyvault/src/ _generated_/rest/core/fetcher/HttpResponsePromise.ts b/skyvault/src/ _generated_/rest/core/fetcher/HttpResponsePromise.ts new file mode 100644 index 00000000..6b0bc4fc --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/HttpResponsePromise.ts @@ -0,0 +1,116 @@ +import { WithRawResponse } from "./RawResponse"; + +/** + * A promise that returns the parsed response and lets you retrieve the raw response too. + */ +export class HttpResponsePromise extends Promise { + private innerPromise: Promise>; + private unwrappedPromise: Promise | undefined; + + private constructor(promise: Promise>) { + // Initialize with a no-op to avoid premature parsing + super((resolve) => { + resolve(undefined as unknown as T); + }); + this.innerPromise = promise; + } + + /** + * Creates an `HttpResponsePromise` from a function that returns a promise. + * + * @param fn - A function that returns a promise resolving to a `WithRawResponse` object. + * @param args - Arguments to pass to the function. + * @returns An `HttpResponsePromise` instance. + */ + public static fromFunction Promise>, T>( + fn: F, + ...args: Parameters + ): HttpResponsePromise { + return new HttpResponsePromise(fn(...args)); + } + + /** + * Creates a function that returns an `HttpResponsePromise` from a function that returns a promise. + * + * @param fn - A function that returns a promise resolving to a `WithRawResponse` object. + * @returns A function that returns an `HttpResponsePromise` instance. + */ + public static interceptFunction< + F extends (...args: never[]) => Promise>, + T = Awaited>["data"], + >(fn: F): (...args: Parameters) => HttpResponsePromise { + return (...args: Parameters): HttpResponsePromise => { + return HttpResponsePromise.fromPromise(fn(...args)); + }; + } + + /** + * Creates an `HttpResponsePromise` from an existing promise. + * + * @param promise - A promise resolving to a `WithRawResponse` object. + * @returns An `HttpResponsePromise` instance. + */ + public static fromPromise(promise: Promise>): HttpResponsePromise { + return new HttpResponsePromise(promise); + } + + /** + * Creates an `HttpResponsePromise` from an executor function. + * + * @param executor - A function that takes resolve and reject callbacks to create a promise. + * @returns An `HttpResponsePromise` instance. + */ + public static fromExecutor( + executor: (resolve: (value: WithRawResponse) => void, reject: (reason?: unknown) => void) => void, + ): HttpResponsePromise { + const promise = new Promise>(executor); + return new HttpResponsePromise(promise); + } + + /** + * Creates an `HttpResponsePromise` from a resolved result. + * + * @param result - A `WithRawResponse` object to resolve immediately. + * @returns An `HttpResponsePromise` instance. + */ + public static fromResult(result: WithRawResponse): HttpResponsePromise { + const promise = Promise.resolve(result); + return new HttpResponsePromise(promise); + } + + private unwrap(): Promise { + if (!this.unwrappedPromise) { + this.unwrappedPromise = this.innerPromise.then(({ data }) => data); + } + return this.unwrappedPromise; + } + + /** @inheritdoc */ + public override then( + onfulfilled?: ((value: T) => TResult1 | PromiseLike) | null, + onrejected?: ((reason: unknown) => TResult2 | PromiseLike) | null, + ): Promise { + return this.unwrap().then(onfulfilled, onrejected); + } + + /** @inheritdoc */ + public override catch( + onrejected?: ((reason: unknown) => TResult | PromiseLike) | null, + ): Promise { + return this.unwrap().catch(onrejected); + } + + /** @inheritdoc */ + public override finally(onfinally?: (() => void) | null): Promise { + return this.unwrap().finally(onfinally); + } + + /** + * Retrieves the data and raw response. + * + * @returns A promise resolving to a `WithRawResponse` object. + */ + public async withRawResponse(): Promise> { + return await this.innerPromise; + } +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/RawResponse.ts b/skyvault/src/ _generated_/rest/core/fetcher/RawResponse.ts new file mode 100644 index 00000000..c975f23c --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/RawResponse.ts @@ -0,0 +1,59 @@ +/** + * The raw response from the fetch call excluding the body. + */ +export type RawResponse = Omit< + { + [K in keyof Response as Response[K] extends Function ? never : K]: Response[K]; // strips out functions + }, + "ok" | "body" | "bodyUsed" +>; // strips out body and bodyUsed + +/** + * A raw response indicating that the request was aborted. + */ +export const abortRawResponse: RawResponse = { + headers: new Headers(), + redirected: false, + status: 499, + statusText: "Client Closed Request", + type: "error", + url: "", +} as const; + +/** + * A raw response indicating an unknown error. + */ +export const unknownRawResponse: RawResponse = { + headers: new Headers(), + redirected: false, + status: 0, + statusText: "Unknown Error", + type: "error", + url: "", +} as const; + +/** + * Converts a `RawResponse` object into a `RawResponse` by extracting its properties, + * excluding the `body` and `bodyUsed` fields. + * + * @param response - The `RawResponse` object to convert. + * @returns A `RawResponse` object containing the extracted properties of the input response. + */ +export function toRawResponse(response: Response): RawResponse { + return { + headers: response.headers, + redirected: response.redirected, + status: response.status, + statusText: response.statusText, + type: response.type, + url: response.url, + }; +} + +/** + * Creates a `RawResponse` from a standard `Response` object. + */ +export interface WithRawResponse { + readonly data: T; + readonly rawResponse: RawResponse; +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/Supplier.ts b/skyvault/src/ _generated_/rest/core/fetcher/Supplier.ts new file mode 100644 index 00000000..867c931c --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/Supplier.ts @@ -0,0 +1,11 @@ +export type Supplier = T | Promise | (() => T | Promise); + +export const Supplier = { + get: async (supplier: Supplier): Promise => { + if (typeof supplier === "function") { + return (supplier as () => T)(); + } else { + return supplier; + } + }, +}; diff --git a/skyvault/src/ _generated_/rest/core/fetcher/createRequestUrl.ts b/skyvault/src/ _generated_/rest/core/fetcher/createRequestUrl.ts new file mode 100644 index 00000000..f1157ce7 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/createRequestUrl.ts @@ -0,0 +1,10 @@ +import qs from "qs"; + +export function createRequestUrl( + baseUrl: string, + queryParameters?: Record, +): string { + return Object.keys(queryParameters ?? {}).length > 0 + ? `${baseUrl}?${qs.stringify(queryParameters, { arrayFormat: "repeat" })}` + : baseUrl; +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/getFetchFn.ts b/skyvault/src/ _generated_/rest/core/fetcher/getFetchFn.ts new file mode 100644 index 00000000..9fd9bfc4 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/getFetchFn.ts @@ -0,0 +1,25 @@ +import { RUNTIME } from "../runtime"; + +/** + * Returns a fetch function based on the runtime + */ +export async function getFetchFn(): Promise { + // In Node.js 18+ environments, use native fetch + if (RUNTIME.type === "node" && RUNTIME.parsedVersion != null && RUNTIME.parsedVersion >= 18) { + return fetch; + } + + // In Node.js 18 or lower environments, the SDK always uses`node-fetch`. + if (RUNTIME.type === "node") { + return (await import("node-fetch")).default as any; + } + + // Otherwise the SDK uses global fetch if available, + // and falls back to node-fetch. + if (typeof fetch == "function") { + return fetch; + } + + // Defaults to node `node-fetch` if global fetch isn't available + return (await import("node-fetch")).default as any; +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/getHeader.ts b/skyvault/src/ _generated_/rest/core/fetcher/getHeader.ts new file mode 100644 index 00000000..50f922b0 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/getHeader.ts @@ -0,0 +1,8 @@ +export function getHeader(headers: Record, header: string): string | undefined { + for (const [headerKey, headerValue] of Object.entries(headers)) { + if (headerKey.toLowerCase() === header.toLowerCase()) { + return headerValue; + } + } + return undefined; +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/getRequestBody.ts b/skyvault/src/ _generated_/rest/core/fetcher/getRequestBody.ts new file mode 100644 index 00000000..fce5589c --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/getRequestBody.ts @@ -0,0 +1,16 @@ +import { toJson } from "../json"; + +export declare namespace GetRequestBody { + interface Args { + body: unknown; + type: "json" | "file" | "bytes" | "other"; + } +} + +export async function getRequestBody({ body, type }: GetRequestBody.Args): Promise { + if (type.includes("json")) { + return toJson(body); + } else { + return body as BodyInit; + } +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/getResponseBody.ts b/skyvault/src/ _generated_/rest/core/fetcher/getResponseBody.ts new file mode 100644 index 00000000..d046e6ea --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/getResponseBody.ts @@ -0,0 +1,34 @@ +import { chooseStreamWrapper } from "./stream-wrappers/chooseStreamWrapper"; + +export async function getResponseBody(response: Response, responseType?: string): Promise { + if (response.body != null && responseType === "blob") { + return await response.blob(); + } else if (response.body != null && responseType === "arrayBuffer") { + return await response.arrayBuffer(); + } else if (response.body != null && responseType === "sse") { + return response.body; + } else if (response.body != null && responseType === "streaming") { + return chooseStreamWrapper(response.body); + } else if (response.body != null && responseType === "text") { + return await response.text(); + } else { + const text = await response.text(); + if (text.length > 0) { + try { + let responseBody = JSON.parse(text); + return responseBody; + } catch (err) { + return { + ok: false, + error: { + reason: "non-json", + statusCode: response.status, + rawBody: text, + }, + }; + } + } else { + return undefined; + } + } +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/index.ts b/skyvault/src/ _generated_/rest/core/fetcher/index.ts new file mode 100644 index 00000000..249f5176 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/index.ts @@ -0,0 +1,8 @@ +export type { APIResponse } from "./APIResponse"; +export { fetcher } from "./Fetcher"; +export type { Fetcher, FetchFunction } from "./Fetcher"; +export { getHeader } from "./getHeader"; +export { Supplier } from "./Supplier"; +export { abortRawResponse, toRawResponse, unknownRawResponse } from "./RawResponse"; +export type { RawResponse, WithRawResponse } from "./RawResponse"; +export { HttpResponsePromise } from "./HttpResponsePromise"; diff --git a/skyvault/src/ _generated_/rest/core/fetcher/makeRequest.ts b/skyvault/src/ _generated_/rest/core/fetcher/makeRequest.ts new file mode 100644 index 00000000..1af42bb9 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/makeRequest.ts @@ -0,0 +1,44 @@ +import { anySignal, getTimeoutSignal } from "./signals"; + +export const makeRequest = async ( + fetchFn: (url: string, init: RequestInit) => Promise, + url: string, + method: string, + headers: Record, + requestBody: BodyInit | undefined, + timeoutMs?: number, + abortSignal?: AbortSignal, + withCredentials?: boolean, + duplex?: "half", +): Promise => { + const signals: AbortSignal[] = []; + + // Add timeout signal + let timeoutAbortId: NodeJS.Timeout | undefined = undefined; + if (timeoutMs != null) { + const { signal, abortId } = getTimeoutSignal(timeoutMs); + timeoutAbortId = abortId; + signals.push(signal); + } + + // Add arbitrary signal + if (abortSignal != null) { + signals.push(abortSignal); + } + let newSignals = anySignal(signals); + const response = await fetchFn(url, { + method: method, + headers, + body: requestBody, + signal: newSignals, + credentials: withCredentials ? "include" : undefined, + // @ts-ignore + duplex, + }); + + if (timeoutAbortId != null) { + clearTimeout(timeoutAbortId); + } + + return response; +}; diff --git a/skyvault/src/ _generated_/rest/core/fetcher/requestWithRetries.ts b/skyvault/src/ _generated_/rest/core/fetcher/requestWithRetries.ts new file mode 100644 index 00000000..add3cce0 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/requestWithRetries.ts @@ -0,0 +1,33 @@ +const INITIAL_RETRY_DELAY = 1000; // in milliseconds +const MAX_RETRY_DELAY = 60000; // in milliseconds +const DEFAULT_MAX_RETRIES = 2; +const JITTER_FACTOR = 0.2; // 20% random jitter + +function addJitter(delay: number): number { + // Generate a random value between -JITTER_FACTOR and +JITTER_FACTOR + const jitterMultiplier = 1 + (Math.random() * 2 - 1) * JITTER_FACTOR; + return delay * jitterMultiplier; +} + +export async function requestWithRetries( + requestFn: () => Promise, + maxRetries: number = DEFAULT_MAX_RETRIES, +): Promise { + let response: Response = await requestFn(); + + for (let i = 0; i < maxRetries; ++i) { + if ([408, 429].includes(response.status) || response.status >= 500) { + // Calculate base delay using exponential backoff (in milliseconds) + const baseDelay = Math.min(INITIAL_RETRY_DELAY * Math.pow(2, i), MAX_RETRY_DELAY); + + // Add jitter to the delay + const delayWithJitter = addJitter(baseDelay); + + await new Promise((resolve) => setTimeout(resolve, delayWithJitter)); + response = await requestFn(); + } else { + break; + } + } + return response!; +} diff --git a/skyvault/src/ _generated_/rest/core/fetcher/signals.ts b/skyvault/src/ _generated_/rest/core/fetcher/signals.ts new file mode 100644 index 00000000..a8d32a2e --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/signals.ts @@ -0,0 +1,38 @@ +const TIMEOUT = "timeout"; + +export function getTimeoutSignal(timeoutMs: number): { signal: AbortSignal; abortId: NodeJS.Timeout } { + const controller = new AbortController(); + const abortId = setTimeout(() => controller.abort(TIMEOUT), timeoutMs); + return { signal: controller.signal, abortId }; +} + +/** + * Returns an abort signal that is getting aborted when + * at least one of the specified abort signals is aborted. + * + * Requires at least node.js 18. + */ +export function anySignal(...args: AbortSignal[] | [AbortSignal[]]): AbortSignal { + // Allowing signals to be passed either as array + // of signals or as multiple arguments. + const signals = (args.length === 1 && Array.isArray(args[0]) ? args[0] : args) as AbortSignal[]; + + const controller = new AbortController(); + + for (const signal of signals) { + if (signal.aborted) { + // Exiting early if one of the signals + // is already aborted. + controller.abort((signal as any)?.reason); + break; + } + + // Listening for signals and removing the listeners + // when at least one symbol is aborted. + signal.addEventListener("abort", () => controller.abort((signal as any)?.reason), { + signal: controller.signal, + }); + } + + return controller.signal; +} diff --git a/src/ _generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts b/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts rename to skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/Node18UniversalStreamWrapper.ts diff --git a/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts b/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts new file mode 100644 index 00000000..23c01a1a --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/NodePre18StreamWrapper.ts @@ -0,0 +1,107 @@ +import type { Readable, Writable } from "readable-stream"; + +import { EventCallback, StreamWrapper } from "./chooseStreamWrapper"; + +export class NodePre18StreamWrapper implements StreamWrapper { + private readableStream: Readable; + private encoding: string | undefined; + + constructor(readableStream: Readable) { + this.readableStream = readableStream; + } + + public on(event: string, callback: EventCallback): void { + this.readableStream.on(event, callback); + } + + public off(event: string, callback: EventCallback): void { + this.readableStream.off(event, callback); + } + + public pipe(dest: Writable): Writable { + this.readableStream.pipe(dest); + return dest; + } + + public pipeTo(dest: Writable): Writable { + return this.pipe(dest); + } + + public unpipe(dest?: Writable): void { + if (dest) { + this.readableStream.unpipe(dest); + } else { + this.readableStream.unpipe(); + } + } + + public destroy(error?: Error): void { + this.readableStream.destroy(error); + } + + public pause(): void { + this.readableStream.pause(); + } + + public resume(): void { + this.readableStream.resume(); + } + + public get isPaused(): boolean { + return this.readableStream.isPaused(); + } + + public async read(): Promise { + return new Promise((resolve, reject) => { + const chunk = this.readableStream.read(); + if (chunk) { + resolve(chunk); + } else { + this.readableStream.once("readable", () => { + const chunk = this.readableStream.read(); + resolve(chunk); + }); + this.readableStream.once("error", reject); + } + }); + } + + public setEncoding(encoding?: string): void { + this.readableStream.setEncoding(encoding as BufferEncoding); + this.encoding = encoding; + } + + public async text(): Promise { + const chunks: Uint8Array[] = []; + const encoder = new TextEncoder(); + this.readableStream.setEncoding((this.encoding || "utf-8") as BufferEncoding); + + for await (const chunk of this.readableStream) { + chunks.push(encoder.encode(chunk)); + } + + const decoder = new TextDecoder(this.encoding || "utf-8"); + return decoder.decode(Buffer.concat(chunks)); + } + + public async json(): Promise { + const text = await this.text(); + return JSON.parse(text); + } + + public [Symbol.asyncIterator](): AsyncIterableIterator { + const readableStream = this.readableStream; + const iterator = readableStream[Symbol.asyncIterator](); + + // Create and return an async iterator that yields buffers + return { + async next(): Promise> { + const { value, done } = await iterator.next(); + return { value: value as Buffer, done }; + }, + [Symbol.asyncIterator]() { + return this; + }, + }; + } +} diff --git a/src/ _generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts b/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts similarity index 100% rename from src/ _generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts rename to skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/UndiciStreamWrapper.ts diff --git a/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts b/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts new file mode 100644 index 00000000..8c7492fc --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/fetcher/stream-wrappers/chooseStreamWrapper.ts @@ -0,0 +1,34 @@ +import type { Readable } from "readable-stream"; + +import { RUNTIME } from "../../runtime"; + +export type EventCallback = (data?: any) => void; + +export interface StreamWrapper { + setEncoding(encoding?: string): void; + on(event: string, callback: EventCallback): void; + off(event: string, callback: EventCallback): void; + pipe(dest: WritableStream): WritableStream; + pipeTo(dest: WritableStream): WritableStream; + unpipe(dest?: WritableStream): void; + destroy(error?: Error): void; + pause(): void; + resume(): void; + get isPaused(): boolean; + read(): Promise; + text(): Promise; + json(): Promise; + [Symbol.asyncIterator](): AsyncIterableIterator; +} + +export async function chooseStreamWrapper(responseBody: any): Promise>> { + if (RUNTIME.type === "node" && RUNTIME.parsedVersion != null && RUNTIME.parsedVersion >= 18) { + return new (await import("./Node18UniversalStreamWrapper")).Node18UniversalStreamWrapper( + responseBody as ReadableStream, + ); + } else if (RUNTIME.type !== "node" && typeof fetch === "function") { + return new (await import("./UndiciStreamWrapper")).UndiciStreamWrapper(responseBody as ReadableStream); + } else { + return new (await import("./NodePre18StreamWrapper")).NodePre18StreamWrapper(responseBody as Readable); + } +} diff --git a/src/ _generated_/rest/core/form-data-utils/FormDataWrapper.ts b/skyvault/src/ _generated_/rest/core/form-data-utils/FormDataWrapper.ts similarity index 100% rename from src/ _generated_/rest/core/form-data-utils/FormDataWrapper.ts rename to skyvault/src/ _generated_/rest/core/form-data-utils/FormDataWrapper.ts diff --git a/src/ _generated_/rest/core/form-data-utils/encodeAsFormParameter.ts b/skyvault/src/ _generated_/rest/core/form-data-utils/encodeAsFormParameter.ts similarity index 100% rename from src/ _generated_/rest/core/form-data-utils/encodeAsFormParameter.ts rename to skyvault/src/ _generated_/rest/core/form-data-utils/encodeAsFormParameter.ts diff --git a/src/ _generated_/rest/core/form-data-utils/index.ts b/skyvault/src/ _generated_/rest/core/form-data-utils/index.ts similarity index 100% rename from src/ _generated_/rest/core/form-data-utils/index.ts rename to skyvault/src/ _generated_/rest/core/form-data-utils/index.ts diff --git a/src/ _generated_/rest/core/index.ts b/skyvault/src/ _generated_/rest/core/index.ts similarity index 100% rename from src/ _generated_/rest/core/index.ts rename to skyvault/src/ _generated_/rest/core/index.ts diff --git a/skyvault/src/ _generated_/rest/core/json.ts b/skyvault/src/ _generated_/rest/core/json.ts new file mode 100644 index 00000000..c052f324 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/json.ts @@ -0,0 +1,27 @@ +/** + * Serialize a value to JSON + * @param value A JavaScript value, usually an object or array, to be converted. + * @param replacer A function that transforms the results. + * @param space Adds indentation, white space, and line break characters to the return-value JSON text to make it easier to read. + * @returns JSON string + */ +export const toJson = ( + value: unknown, + replacer?: (this: unknown, key: string, value: unknown) => unknown, + space?: string | number, +): string => { + return JSON.stringify(value, replacer, space); +}; + +/** + * Parse JSON string to object, array, or other type + * @param text A valid JSON string. + * @param reviver A function that transforms the results. This function is called for each member of the object. If a member contains nested objects, the nested objects are transformed before the parent object is. + * @returns Parsed object, array, or other type + */ +export function fromJson( + text: string, + reviver?: (this: unknown, key: string, value: unknown) => unknown, +): T { + return JSON.parse(text, reviver); +} diff --git a/skyvault/src/ _generated_/rest/core/runtime/index.ts b/skyvault/src/ _generated_/rest/core/runtime/index.ts new file mode 100644 index 00000000..5c76dbb1 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/runtime/index.ts @@ -0,0 +1 @@ +export { RUNTIME } from "./runtime"; diff --git a/skyvault/src/ _generated_/rest/core/runtime/runtime.ts b/skyvault/src/ _generated_/rest/core/runtime/runtime.ts new file mode 100644 index 00000000..a9750175 --- /dev/null +++ b/skyvault/src/ _generated_/rest/core/runtime/runtime.ts @@ -0,0 +1,131 @@ +interface DenoGlobal { + version: { + deno: string; + }; +} + +interface BunGlobal { + version: string; +} + +declare const Deno: DenoGlobal | undefined; +declare const Bun: BunGlobal | undefined; +declare const EdgeRuntime: string | undefined; + +/** + * A constant that indicates which environment and version the SDK is running in. + */ +export const RUNTIME: Runtime = evaluateRuntime(); + +export interface Runtime { + type: "browser" | "web-worker" | "deno" | "bun" | "node" | "react-native" | "unknown" | "workerd" | "edge-runtime"; + version?: string; + parsedVersion?: number; +} + +function evaluateRuntime(): Runtime { + /** + * A constant that indicates whether the environment the code is running is a Web Browser. + */ + const isBrowser = typeof window !== "undefined" && typeof window.document !== "undefined"; + if (isBrowser) { + return { + type: "browser", + version: window.navigator.userAgent, + }; + } + + /** + * A constant that indicates whether the environment the code is running is Cloudflare. + * https://developers.cloudflare.com/workers/runtime-apis/web-standards/#navigatoruseragent + */ + const isCloudflare = typeof globalThis !== "undefined" && globalThis?.navigator?.userAgent === "Cloudflare-Workers"; + if (isCloudflare) { + return { + type: "workerd", + }; + } + + /** + * A constant that indicates whether the environment the code is running is Edge Runtime. + * https://vercel.com/docs/functions/runtimes/edge-runtime#check-if-you're-running-on-the-edge-runtime + */ + const isEdgeRuntime = typeof EdgeRuntime === "string"; + if (isEdgeRuntime) { + return { + type: "edge-runtime", + }; + } + + /** + * A constant that indicates whether the environment the code is running is a Web Worker. + */ + const isWebWorker = + typeof self === "object" && + // @ts-ignore + typeof self?.importScripts === "function" && + (self.constructor?.name === "DedicatedWorkerGlobalScope" || + self.constructor?.name === "ServiceWorkerGlobalScope" || + self.constructor?.name === "SharedWorkerGlobalScope"); + if (isWebWorker) { + return { + type: "web-worker", + }; + } + + /** + * A constant that indicates whether the environment the code is running is Deno. + * FYI Deno spoofs process.versions.node, see https://deno.land/std@0.177.0/node/process.ts?s=versions + */ + const isDeno = + typeof Deno !== "undefined" && typeof Deno.version !== "undefined" && typeof Deno.version.deno !== "undefined"; + if (isDeno) { + return { + type: "deno", + version: Deno.version.deno, + }; + } + + /** + * A constant that indicates whether the environment the code is running is Bun.sh. + */ + const isBun = typeof Bun !== "undefined" && typeof Bun.version !== "undefined"; + if (isBun) { + return { + type: "bun", + version: Bun.version, + }; + } + + /** + * A constant that indicates whether the environment the code is running is Node.JS. + */ + const isNode = + typeof process !== "undefined" && + "version" in process && + !!process.version && + "versions" in process && + !!process.versions?.node; + if (isNode) { + return { + type: "node", + version: process.versions.node, + parsedVersion: Number(process.versions.node.split(".")[0]), + }; + } + + /** + * A constant that indicates whether the environment the code is running is in React-Native. + * https://github.com/facebook/react-native/blob/main/packages/react-native/Libraries/Core/setUpNavigator.js + */ + const isReactNative = typeof navigator !== "undefined" && navigator?.product === "ReactNative"; + if (isReactNative) { + return { + type: "react-native", + }; + } + + return { + type: "unknown", + }; +} diff --git a/src/ _generated_/rest/environments.ts b/skyvault/src/ _generated_/rest/environments.ts similarity index 100% rename from src/ _generated_/rest/environments.ts rename to skyvault/src/ _generated_/rest/environments.ts diff --git a/skyvault/src/ _generated_/rest/errors/SkyflowError.ts b/skyvault/src/ _generated_/rest/errors/SkyflowError.ts new file mode 100644 index 00000000..003768c2 --- /dev/null +++ b/skyvault/src/ _generated_/rest/errors/SkyflowError.ts @@ -0,0 +1,55 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as core from "../core"; +import { toJson } from "../core/json"; + +export class SkyflowError extends Error { + public readonly statusCode?: number; + public readonly body?: unknown; + public readonly rawResponse?: core.RawResponse; + + constructor({ + message, + statusCode, + body, + rawResponse, + }: { + message?: string; + statusCode?: number; + body?: unknown; + rawResponse?: core.RawResponse; + }) { + super(buildMessage({ message, statusCode, body })); + Object.setPrototypeOf(this, SkyflowError.prototype); + this.statusCode = statusCode; + this.body = body; + this.rawResponse = rawResponse; + } +} + +function buildMessage({ + message, + statusCode, + body, +}: { + message: string | undefined; + statusCode: number | undefined; + body: unknown | undefined; +}): string { + let lines: string[] = []; + if (message != null) { + lines.push(message); + } + + if (statusCode != null) { + lines.push(`Status code: ${statusCode.toString()}`); + } + + if (body != null) { + lines.push(`Body: ${toJson(body, undefined, 2)}`); + } + + return lines.join("\n"); +} diff --git a/skyvault/src/ _generated_/rest/errors/SkyflowTimeoutError.ts b/skyvault/src/ _generated_/rest/errors/SkyflowTimeoutError.ts new file mode 100644 index 00000000..75024f63 --- /dev/null +++ b/skyvault/src/ _generated_/rest/errors/SkyflowTimeoutError.ts @@ -0,0 +1,10 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export class SkyflowTimeoutError extends Error { + constructor(message: string) { + super(message); + Object.setPrototypeOf(this, SkyflowTimeoutError.prototype); + } +} diff --git a/skyvault/src/ _generated_/rest/errors/index.ts b/skyvault/src/ _generated_/rest/errors/index.ts new file mode 100644 index 00000000..5567f00d --- /dev/null +++ b/skyvault/src/ _generated_/rest/errors/index.ts @@ -0,0 +1,2 @@ +export { SkyflowError } from "./SkyflowError"; +export { SkyflowTimeoutError } from "./SkyflowTimeoutError"; diff --git a/skyvault/src/ _generated_/rest/index.ts b/skyvault/src/ _generated_/rest/index.ts new file mode 100644 index 00000000..0706e084 --- /dev/null +++ b/skyvault/src/ _generated_/rest/index.ts @@ -0,0 +1,4 @@ +export * as Skyflow from "./api"; +export { SkyflowClient } from "./Client"; +export { SkyflowEnvironment } from "./environments"; +export { SkyflowError, SkyflowTimeoutError } from "./errors"; diff --git a/skyvault/src/ _generated_/rest/version.ts b/skyvault/src/ _generated_/rest/version.ts new file mode 100644 index 00000000..e3655fb3 --- /dev/null +++ b/skyvault/src/ _generated_/rest/version.ts @@ -0,0 +1 @@ +export const SDK_VERSION = "1.0.21"; diff --git a/src/index.ts b/skyvault/src/index.ts similarity index 91% rename from src/index.ts rename to skyvault/src/index.ts index a64396c4..a49d2ad2 100644 --- a/src/index.ts +++ b/skyvault/src/index.ts @@ -1,5 +1,5 @@ import Skyflow from './vault/skyflow'; -import { LogLevel, Env, RedactionType, RequestMethod, OrderByEnum, TokenMode, DetectEntities } from './utils'; +import { LogLevel, Env, RedactionType, RequestMethod, OrderByEnum, TokenMode, DetectEntities } from "skyflow-common"; import InsertRequest from './vault/model/request/insert'; import InsertOptions from './vault/model/options/insert'; import GetRequest from './vault/model/request/get'; @@ -11,12 +11,12 @@ import DeleteRequest from './vault/model/request/delete'; import UpdateRequest from './vault/model/request/update'; import FileUploadRequest from './vault/model/request/file-upload'; import QueryRequest from './vault/model/request/query'; -import Credentials from './vault/config/credentials'; -import { TokenCredentials, ApiKeyCredentials, PathCredentials, StringCredentials } from './vault/config/credentials'; +import { Credentials } from "skyflow-common"; +import { TokenCredentials, ApiKeyCredentials, PathCredentials, StringCredentials } from "skyflow-common"; import TokenizeRequest from './vault/model/request/tokenize'; import TokenizeResponse from './vault/model/response/tokenize'; -import { BearerTokenOptions, generateBearerToken, generateBearerTokenFromCreds, generateSignedDataTokens, generateSignedDataTokensFromCreds, GenerateTokenOptions, SignedDataTokensOptions } from './service-account'; -import { isExpired } from './utils/jwt-utils'; +import { BearerTokenOptions, generateBearerToken, generateBearerTokenFromCreds, generateSignedDataTokens, generateSignedDataTokensFromCreds, GenerateTokenOptions, SignedDataTokensOptions } from "skyflow-common"; +import { isExpired } from "skyflow-common"; import UpdateOptions from './vault/model/options/update'; import InvokeConnectionRequest from './vault/model/request/inkove'; import GetColumnRequest from './vault/model/request/get-column'; @@ -29,9 +29,9 @@ import FileUploadResponse from './vault/model/response/file-upload'; import QueryResponse from './vault/model/response/query'; import InvokeConnectionResponse from './vault/model/response/invoke/invoke'; import { SkyflowConfig, TokenizeRequestType, DetokenizeData, InsertResponseType, GetResponseData, QueryResponseType, IndexRange, FileInput, StringKeyValueMapType } from './vault/types'; -import VaultConfig from './vault/config/vault'; -import SkyflowError from './error'; -import ConnectionConfig from './vault/config/connection'; +import { VaultConfig } from "skyflow-common"; +import { SkyflowError } from "skyflow-common"; +import { ConnectionConfig } from "skyflow-common"; import DeidentifyTextRequest from './vault/model/request/deidentify-text'; import DeidentifyTextOptions from './vault/model/options/deidentify-text'; import DeidentifyTextResponse from './vault/model/response/deidentify-text'; @@ -44,9 +44,9 @@ import DeidentifyFileOptions from './vault/model/options/deidentify-file'; import DeidentifyFileRequest from './vault/model/request/deidentify-file'; import DeidentifyFileResponse from './vault/model/response/deidentify-file'; import GetDetectRunRequest from './vault/model/request/get-detect-run'; -import { TokenType, MaskingMethod, DetectOutputTranscription } from './utils'; +import { TokenType, MaskingMethod, DetectOutputTranscription } from "skyflow-common"; import { Bleep } from './vault/model/options/deidentify-file/bleep-audio'; -import { SkyflowRecordError } from './utils/index'; +import { SkyflowRecordError } from "skyflow-common"; export { Env, LogLevel, diff --git a/src/management/config/management/index.ts b/skyvault/src/management/config/management/index.ts similarity index 100% rename from src/management/config/management/index.ts rename to skyvault/src/management/config/management/index.ts diff --git a/src/management/controller/policy/index.ts b/skyvault/src/management/controller/policy/index.ts similarity index 100% rename from src/management/controller/policy/index.ts rename to skyvault/src/management/controller/policy/index.ts diff --git a/src/management/controller/role/index.ts b/skyvault/src/management/controller/role/index.ts similarity index 100% rename from src/management/controller/role/index.ts rename to skyvault/src/management/controller/role/index.ts diff --git a/src/management/controller/vault/index.ts b/skyvault/src/management/controller/vault/index.ts similarity index 100% rename from src/management/controller/vault/index.ts rename to skyvault/src/management/controller/vault/index.ts diff --git a/src/management/index.ts b/skyvault/src/management/index.ts similarity index 100% rename from src/management/index.ts rename to skyvault/src/management/index.ts diff --git a/src/management/model/request/create-vault/index.ts b/skyvault/src/management/model/request/create-vault/index.ts similarity index 100% rename from src/management/model/request/create-vault/index.ts rename to skyvault/src/management/model/request/create-vault/index.ts diff --git a/src/management/model/request/delete-policy/index.ts b/skyvault/src/management/model/request/delete-policy/index.ts similarity index 100% rename from src/management/model/request/delete-policy/index.ts rename to skyvault/src/management/model/request/delete-policy/index.ts diff --git a/src/management/model/response/create-vault/index.ts b/skyvault/src/management/model/response/create-vault/index.ts similarity index 100% rename from src/management/model/response/create-vault/index.ts rename to skyvault/src/management/model/response/create-vault/index.ts diff --git a/src/management/model/response/delete-policy/index.ts b/skyvault/src/management/model/response/delete-policy/index.ts similarity index 100% rename from src/management/model/response/delete-policy/index.ts rename to skyvault/src/management/model/response/delete-policy/index.ts diff --git a/src/management/skyflow/index.ts b/skyvault/src/management/skyflow/index.ts similarity index 100% rename from src/management/skyflow/index.ts rename to skyvault/src/management/skyflow/index.ts diff --git a/src/management/types/index.ts b/skyvault/src/management/types/index.ts similarity index 78% rename from src/management/types/index.ts rename to skyvault/src/management/types/index.ts index 7841e394..2ac72ae7 100644 --- a/src/management/types/index.ts +++ b/skyvault/src/management/types/index.ts @@ -1,4 +1,4 @@ -import { LogLevel } from "../../utils"; +import { LogLevel } from "skyflow-common"; import ManagementConfig from "../config/management"; export interface SkyflowConfig { diff --git a/src/utils/validations/index.ts b/skyvault/src/utils/validations/index.ts similarity index 69% rename from src/utils/validations/index.ts rename to skyvault/src/utils/validations/index.ts index 8123f0ef..f1ae580e 100644 --- a/src/utils/validations/index.ts +++ b/skyvault/src/utils/validations/index.ts @@ -1,10 +1,5 @@ -import { CONFIG, Env, HTTP_HEADER, isValidURL, LogLevel, MessageType, RequestMethod, OrderByEnum, parameterizedString, printLog, RedactionType, SKYFLOW, TokenMode, API_KEY } from ".."; +import { HTTP_HEADER, LogLevel, MessageType, RequestMethod, OrderByEnum, parameterizedString, printLog, RedactionType, SKYFLOW, TokenMode, SkyflowError, SKYFLOW_ERROR_CODE, logs, StringKeyValueMapType } from "skyflow-common"; import { V1Byot } from "../../ _generated_/rest/api"; -import SkyflowError from "../../error"; -import SKYFLOW_ERROR_CODE from "../../error/codes"; -import ConnectionConfig from "../../vault/config/connection"; -import Credentials, { ApiKeyCredentials, PathCredentials, StringCredentials, TokenCredentials } from "../../vault/config/credentials"; -import VaultConfig from "../../vault/config/vault"; import DetokenizeOptions from "../../vault/model/options/detokenize"; import GetOptions from "../../vault/model/options/get"; import InsertOptions from "../../vault/model/options/insert"; @@ -19,11 +14,8 @@ import InsertRequest from "../../vault/model/request/insert"; import QueryRequest from "../../vault/model/request/query"; import TokenizeRequest from "../../vault/model/request/tokenize"; import UpdateRequest from "../../vault/model/request/update"; -import { SkyflowConfig, StringKeyValueMapType } from "../../vault/types"; import * as fs from 'fs'; import * as path from 'path'; -import { isExpired } from "../jwt-utils"; -import logs from "../logs"; import FileUploadOptions from "../../vault/model/options/fileUpload"; import DeidentifyTextRequest from "../../vault/model/request/deidentify-text"; import DeidentifyTextOptions from "../../vault/model/options/deidentify-text"; @@ -36,10 +28,6 @@ import DeidentifyFileRequest from "../../vault/model/request/deidentify-file"; import { Bleep } from "../../vault/model/options/deidentify-file/bleep-audio"; import GetDetectRunRequest from "../../vault/model/request/get-detect-run"; -export function isEnv(value?: string): boolean { - return value !== undefined && Object.values(Env).includes(value as Env); -} - export function isRedactionType(value?: string): boolean { return value !== undefined && Object.values(RedactionType).includes(value as RedactionType); } @@ -56,372 +44,6 @@ export function isMethod(value?: string): boolean { return value !== undefined && Object.values(RequestMethod).includes(value as RequestMethod); } -export function isLogLevel(value?: string): boolean { - return value !== undefined && Object.values(LogLevel).includes(value as LogLevel); -} - -export function isValidAPIKey(apiKey: string) { - if (!apiKey) { - return false; - } - if (apiKey && typeof apiKey === 'string' && apiKey.startsWith(API_KEY.PREFIX)) { - return true; - } - return false; -}; - -function isValidCredentialsString(credentialsString: string) { - if (!credentialsString || credentialsString === null || credentialsString === undefined) { - return false; - } - if (credentialsString && typeof credentialsString === 'string') { - try { - let credentialsObj = JSON.parse("{}") - const parsed = JSON.parse(credentialsString); - credentialsObj = { - ...parsed, - clientId: parsed.clientId ?? parsed.clientID, - keyId: parsed.keyId ?? parsed.keyID, - tokenUri: parsed.tokenUri ?? parsed.tokenURI, - }; - if (credentialsObj?.clientId == null || credentialsObj?.keyId == null) { - return false; - } - return true; - } catch (err) { - return false; - } - } - return false; -}; - -function isValidPath(path: string) { - if (!path || path === null || path === undefined) { - return false; - } - if (path && typeof path === 'string' && fs.existsSync(path)) { - return true; - } - return false; -}; - -export const validateSkyflowConfig = (config: SkyflowConfig, logLevel: LogLevel = LogLevel.ERROR) => { - if (config) { - if (!Object.prototype.hasOwnProperty.call(config, 'vaultConfigs')) { - printLog(logs.errorLogs.VAULT_CONFIG_KEY_MISSING, MessageType.ERROR, logLevel); - } - - const { vaultConfigs, connectionConfigs } = config; - - // Count how many of the fields are defined - const definedFields = [vaultConfigs, connectionConfigs].filter(Boolean).length; - - // If none are present - if (definedFields === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_SKYFLOW_CONFIG); - } - - if (config?.vaultConfigs && !Array.isArray(config.vaultConfigs)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TYPE_FOR_CONFIG, [CONFIG.VAULT]) - } - - if (config?.connectionConfigs && !Array.isArray(config?.connectionConfigs)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TYPE_FOR_CONFIG, [CONFIG.CONNECTION]) - } - - } else { - throw new SkyflowError(SKYFLOW_ERROR_CODE.CONFIG_MISSING); - } -}; - - -export const validateCredentialsWithId = (credentials: Credentials, type: string, typeId: string, id: string, logLevel: LogLevel = LogLevel.ERROR) => { - if (!credentials) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CREDENTIALS_WITH_ID, [type, typeId, id]); - } - - const isTokenCred = 'token' in credentials; - const isPathCred = 'path' in credentials; - const isStringCred = 'credentialsString' in credentials; - const isApiKeyCred = 'apiKey' in credentials; - - // Check if exactly one credential type is provided - const definedTypes = [isTokenCred, isPathCred, isStringCred, isApiKeyCred].filter(Boolean).length; - - if (definedTypes === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CREDENTIALS_WITH_ID, [type, typeId, id]); - } - - if (definedTypes > 1) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.MULTIPLE_CREDENTIALS_PASSED_WITH_ID, [type, typeId, id]); - } - - // Validate TokenCredentials - if (isTokenCred) { - const tokenCred = credentials as TokenCredentials; - if (typeof tokenCred.token !== 'string' || isExpired(tokenCred.token)) { - printLog(logs.errorLogs.EMPTY_TOKEN_VALUE, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_BEARER_TOKEN_WITH_ID, [type, typeId, id]); - } - } - - // Validate PathCredentials - if (isPathCred) { - const pathCred = credentials as PathCredentials; - if (typeof pathCred.path !== 'string' || !isValidPath(pathCred.path)) { - printLog(logs.errorLogs.EMPTY_CREDENTIALS_PATH, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_FILE_PATH_WITH_ID, [type, typeId, id]); - } - if (pathCred.roles !== undefined && !Array.isArray(pathCred.roles)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE, [type, typeId, id]); - } - if (pathCred.context !== undefined && (typeof pathCred.context !== 'string' && typeof pathCred.context !== 'object')) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT, [type, typeId, id]); - } - if(Object.prototype.hasOwnProperty.call(pathCred, 'tokenUri')) { - if (pathCred.tokenUri === undefined || typeof pathCred.tokenUri !== 'string' || !isValidURL(pathCred.tokenUri)) { - printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI_WITH_ID, [type, typeId, id]); - } - } - } - - // Validate StringCredentials - if (isStringCred) { - const stringCred = credentials as StringCredentials; - if (typeof stringCred.credentialsString !== 'string' || !isValidCredentialsString(stringCred.credentialsString)) { - printLog(logs.errorLogs.EMPTY_CREDENTIALS_STRING, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_PARSED_CREDENTIALS_STRING_WITH_ID, [type, typeId, id]); - } - if (stringCred.roles !== undefined && !Array.isArray(stringCred.roles)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE, [type, typeId, id]); - } - if (stringCred.context !== undefined && (typeof stringCred.context !== 'string' && typeof stringCred.context !== 'object')) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT, [type, typeId, id]); - } - if (Object.prototype.hasOwnProperty.call(stringCred, 'tokenUri')) { - if (stringCred.tokenUri === undefined || typeof stringCred.tokenUri !== 'string' || !isValidURL(stringCred.tokenUri)) { - printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI_WITH_ID, [type, typeId, id]); - } - } - } - - // Validate ApiKeyCredentials - if (isApiKeyCred) { - const apiKeyCred = credentials as ApiKeyCredentials; - if (typeof apiKeyCred.apiKey !== 'string' || !isValidAPIKey(apiKeyCred.apiKey)) { - printLog(logs.errorLogs.INVALID_KEY, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_KEY_WITH_ID, [type, typeId, id]); - } - } -}; - -export const validateVaultConfig = (vaultConfig: VaultConfig, logLevel: LogLevel = LogLevel.ERROR) => { - if (vaultConfig) { - if (!Object.prototype.hasOwnProperty.call(vaultConfig, 'vaultId')) { - printLog(logs.errorLogs.EMPTY_VAULT_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_ID); - } - if (!vaultConfig.vaultId || typeof vaultConfig.vaultId !== 'string') { - printLog(logs.errorLogs.INVALID_VAULT_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_VAULT_ID); - } - if (!Object.prototype.hasOwnProperty.call(vaultConfig, 'clusterId')) { - printLog(logs.errorLogs.EMPTY_CLUSTER_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CLUSTER_ID, [vaultConfig?.vaultId]); - } - if (!vaultConfig.clusterId || typeof vaultConfig.clusterId !== 'string') { - printLog(logs.errorLogs.INVALID_CLUSTER_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CLUSTER_ID, [vaultConfig?.vaultId]); - } - if (vaultConfig?.env && !isEnv(vaultConfig.env)) { - printLog(logs.errorLogs.INVALID_ENV, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ENV, [vaultConfig?.vaultId]); - } - if (vaultConfig?.credentials) { - validateCredentialsWithId(vaultConfig.credentials, CONFIG.VAULT, CONFIG.VAULT_ID, vaultConfig.vaultId, logLevel); - } - } else { - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_CONFIG); - } -}; - -export const validateUpdateVaultConfig = (vaultConfig: VaultConfig, logLevel: LogLevel = LogLevel.ERROR) => { - if (vaultConfig) { - if (!Object.prototype.hasOwnProperty.call(vaultConfig, 'vaultId')) { - printLog(logs.errorLogs.EMPTY_VAULT_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_ID); - } - if (!vaultConfig?.vaultId || typeof vaultConfig?.vaultId !== 'string') { - printLog(logs.errorLogs.INVALID_VAULT_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_VAULT_ID); - } - if (vaultConfig?.clusterId && typeof vaultConfig.clusterId !== 'string') { - printLog(logs.errorLogs.INVALID_CLUSTER_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CLUSTER_ID, [vaultConfig?.vaultId]); - } - if (vaultConfig?.env && !isEnv(vaultConfig.env)) { - printLog(logs.errorLogs.INVALID_ENV, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ENV, [vaultConfig?.vaultId]); - } - if (vaultConfig?.credentials) { - validateCredentialsWithId(vaultConfig.credentials, CONFIG.VAULT, CONFIG.VAULT_ID, vaultConfig.vaultId, logLevel); - } - } else { - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_VAULT_CONFIG); - } -}; - -export const validateSkyflowCredentials = (credentials: Credentials, logLevel: LogLevel = LogLevel.ERROR) => { - if (!credentials) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.CREDENTIALS_WITH_NO_VALID_KEY); - } - - const isTokenCred = 'token' in credentials; - const isPathCred = 'path' in credentials; - const isStringCred = 'credentialsString' in credentials; - const isApiKeyCred = 'apiKey' in credentials; - - // Check if exactly one credential type is provided - const definedTypes = [isTokenCred, isPathCred, isStringCred, isApiKeyCred].filter(Boolean).length; - - if (definedTypes === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.CREDENTIALS_WITH_NO_VALID_KEY); - } - - if (definedTypes > 1) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.MULTIPLE_CREDENTIALS_PASSED); - } - - // Validate TokenCredentials - if (isTokenCred) { - const tokenCred = credentials as TokenCredentials; - if (typeof tokenCred.token !== 'string' || isExpired(tokenCred.token)) { - printLog(logs.errorLogs.EMPTY_TOKEN_VALUE, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_BEARER_TOKEN); - } - } - - // Validate PathCredentials - if (isPathCred) { - const pathCred = credentials as PathCredentials; - if (typeof pathCred.path !== 'string' || !isValidPath(pathCred.path)) { - printLog(logs.errorLogs.EMPTY_CREDENTIALS_PATH, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CREDENTIALS_FILE_PATH); - } - if (pathCred.roles !== undefined && !Array.isArray(pathCred.roles)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE); - } - if (pathCred.context !== undefined && (typeof pathCred.context !== 'string' && typeof pathCred.context !== 'object')) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT); - } - - if(Object.prototype.hasOwnProperty.call(pathCred, 'tokenUri')) { - if (pathCred.tokenUri === undefined || typeof pathCred.tokenUri !== 'string' || !isValidURL(pathCred.tokenUri)) { - printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI); - } - } - } - - // Validate StringCredentials - if (isStringCred) { - const stringCred = credentials as StringCredentials; - if (typeof stringCred.credentialsString !== 'string' || !isValidCredentialsString(stringCred.credentialsString)) { - printLog(logs.errorLogs.EMPTY_CREDENTIALS_STRING, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_PARSED_CREDENTIALS_STRING); - } - if (stringCred.roles !== undefined && !Array.isArray(stringCred.roles)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ROLES_KEY_TYPE); - } - // validate both string | Record - if (stringCred.context !== undefined && (typeof stringCred.context !== 'string' && typeof stringCred.context !== 'object')) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONTEXT); - } - if (Object.prototype.hasOwnProperty.call(stringCred, 'tokenUri')) { - if (stringCred.tokenUri === undefined || typeof stringCred.tokenUri !== 'string' || !isValidURL(stringCred.tokenUri)) { - printLog(logs.errorLogs.INVALID_TOKEN_URI, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_URI); - } - } - } - - // Validate ApiKeyCredentials - if (isApiKeyCred) { - const apiKeyCred = credentials as ApiKeyCredentials; - if (typeof apiKeyCred.apiKey !== 'string' || !isValidAPIKey(apiKeyCred.apiKey)) { - printLog(logs.errorLogs.INVALID_KEY, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_KEY); - } - } -}; - -export const validateConnectionConfig = (connectionConfig: ConnectionConfig, logLevel: LogLevel = LogLevel.ERROR) => { - if (connectionConfig) { - if (!Object.prototype.hasOwnProperty.call(connectionConfig, 'connectionId')) { - printLog(logs.errorLogs.EMPTY_CONNECTION_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_ID); - } - - if (typeof connectionConfig?.connectionId !== 'string' || connectionConfig?.connectionId.trim().length === 0) { - printLog(logs.errorLogs.INVALID_CONNECTION_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_ID); - } - - if (!Object.prototype.hasOwnProperty.call(connectionConfig, 'connectionUrl')) { - printLog(logs.errorLogs.EMPTY_CONNECTION_URL, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_URL); - } - - if (typeof connectionConfig?.connectionUrl !== 'string' || connectionConfig?.connectionUrl.trim().length === 0) { - printLog(logs.errorLogs.INVALID_CONNECTION_URL, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); - } - - if (connectionConfig.connectionUrl && !isValidURL(connectionConfig.connectionUrl)) { - printLog(logs.errorLogs.INVALID_CONNECTION_URL_TYPE, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); - } - - if (connectionConfig?.credentials) { - validateCredentialsWithId(connectionConfig.credentials, CONFIG.CONNECTION, CONFIG.CONNECTION_ID, connectionConfig.connectionId, logLevel); - } - } else { - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_CONFIG) - } -}; - -export const validateUpdateConnectionConfig = (connectionConfig: ConnectionConfig, logLevel: LogLevel = LogLevel.ERROR) => { - if (connectionConfig) { - if (!Object.prototype.hasOwnProperty.call(connectionConfig, 'connectionId')) { - printLog(logs.errorLogs.EMPTY_CONNECTION_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_ID); - } - - if (typeof connectionConfig?.connectionId !== 'string' || connectionConfig?.connectionId.trim().length === 0) { - printLog(logs.errorLogs.INVALID_CONNECTION_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_ID); - } - - if (connectionConfig?.connectionUrl && (typeof connectionConfig?.connectionUrl !== 'string' || connectionConfig?.connectionUrl.trim().length === 0)) { - printLog(logs.errorLogs.INVALID_CONNECTION_URL, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); - } - - if (connectionConfig?.connectionUrl && !isValidURL(connectionConfig.connectionUrl)) { - printLog(logs.errorLogs.INVALID_CONNECTION_URL_TYPE, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_CONNECTION_URL); - } - - if (connectionConfig?.credentials) { - validateCredentialsWithId(connectionConfig.credentials, CONFIG.CONNECTION, CONFIG.CONNECTION_ID, connectionConfig.connectionId, logLevel); - } - } else { - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_CONFIG) - } -}; - function validateInsertInput(input: unknown, index: number): void { if (typeof input !== 'object' || input === null || Array.isArray(input)) { throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_RECORD_IN_INSERT, [index]); diff --git a/src/vault/client/index.ts b/skyvault/src/vault/client/index.ts similarity index 98% rename from src/vault/client/index.ts rename to skyvault/src/vault/client/index.ts index 29f856fd..7bc5946d 100644 --- a/src/vault/client/index.ts +++ b/skyvault/src/vault/client/index.ts @@ -4,12 +4,12 @@ import { Query } from "../../ _generated_/rest/api/resources/query/client/Client import { Records } from "../../ _generated_/rest/api/resources/records/client/Client"; import { Strings } from "../../ _generated_/rest/api/resources/strings/client/Client"; import { Tokens } from "../../ _generated_/rest/api/resources/tokens/client/Client"; -import SkyflowError from "../../error"; -import errorMessages from "../../error/messages"; -import { AuthInfo, AuthType, LogLevel, MessageType, printLog, TYPES, HTTP_HEADER, CONTENT_TYPE, BOOLEAN_STRING, HTTP_STATUS_CODE } from "../../utils/index"; -import { isExpired } from "../../utils/jwt-utils"; -import logs from "../../utils/logs"; -import Credentials from "../config/credentials"; +import { SkyflowError } from "skyflow-common"; +import { errorMessages } from "skyflow-common"; +import { AuthInfo, AuthType, LogLevel, MessageType, printLog, TYPES, HTTP_HEADER, CONTENT_TYPE, BOOLEAN_STRING, HTTP_STATUS_CODE } from "skyflow-common"; +import { isExpired } from "skyflow-common"; +import { logs } from "skyflow-common"; +import { Credentials } from "skyflow-common"; import { SkyflowApiErrorLegacy, SkyflowApiErrorNewFormat, SkyflowErrorData } from "../types"; class VaultClient { diff --git a/src/vault/controller/audit/index.ts b/skyvault/src/vault/controller/audit/index.ts similarity index 100% rename from src/vault/controller/audit/index.ts rename to skyvault/src/vault/controller/audit/index.ts diff --git a/src/vault/controller/binlookup/index.ts b/skyvault/src/vault/controller/binlookup/index.ts similarity index 100% rename from src/vault/controller/binlookup/index.ts rename to skyvault/src/vault/controller/binlookup/index.ts diff --git a/src/vault/controller/connections/index.ts b/skyvault/src/vault/controller/connections/index.ts similarity index 98% rename from src/vault/controller/connections/index.ts rename to skyvault/src/vault/controller/connections/index.ts index 0c3025fc..8f6b863b 100644 --- a/src/vault/controller/connections/index.ts +++ b/skyvault/src/vault/controller/connections/index.ts @@ -15,14 +15,14 @@ import { HTTP_HEADER, CONTENT_TYPE, objectToXML, -} from "../../../utils"; +} from "skyflow-common"; import InvokeConnectionRequest from "../../model/request/inkove"; -import logs from "../../../utils/logs"; +import { logs } from "skyflow-common"; import { validateInvokeConnectionRequest } from "../../../utils/validations"; import VaultClient from "../../client"; import InvokeConnectionResponse from "../../model/response/invoke/invoke"; -import SkyflowError from "../../../error"; -import SKYFLOW_ERROR_CODE from "../../../error/codes"; +import { SkyflowError } from "skyflow-common"; +import { SKYFLOW_ERROR_CODE } from "skyflow-common"; class ConnectionController { private client: VaultClient; diff --git a/src/vault/controller/detect/index.ts b/skyvault/src/vault/controller/detect/index.ts similarity index 99% rename from src/vault/controller/detect/index.ts rename to skyvault/src/vault/controller/detect/index.ts index 682b83f6..b1425a6f 100644 --- a/src/vault/controller/detect/index.ts +++ b/skyvault/src/vault/controller/detect/index.ts @@ -5,8 +5,8 @@ import { DeidentifyTextRequest as DeidentifyTextRequest2,DeidentifyAudioRequest, import { DeidentifyFileRequest as DeidentifyFileRequest2} from "../../../ _generated_/rest/api"; import { TokenType } from "../../../ _generated_/rest/api"; -import { DeidenitfyFileRequestTypes, generateSDKMetrics, getBearerToken, MessageType, parameterizedString, printLog, removeSDKVersion, SDK, TYPES, HTTP_HEADER, DETECT_STATUS, FILE_EXTENSION, FILE_FORMAT_TYPE, FILE_PROCESSING, ENCODING_TYPE } from "../../../utils"; -import logs from "../../../utils/logs"; +import { DeidenitfyFileRequestTypes, generateSDKMetrics, getBearerToken, MessageType, parameterizedString, printLog, removeSDKVersion, SDK, TYPES, HTTP_HEADER, DETECT_STATUS, FILE_EXTENSION, FILE_FORMAT_TYPE, FILE_PROCESSING, ENCODING_TYPE } from "skyflow-common"; +import { logs } from "skyflow-common"; import { validateDeIdentifyTextRequest, validateReidentifyTextRequest, validateDeidentifyFileRequest, validateGetDetectRunRequest } from "../../../utils/validations"; import VaultClient from "../../client"; import DeidentifyTextOptions from "../../model/options/deidentify-text"; @@ -20,8 +20,8 @@ import DeidentifyFileOptions from "../../model/options/deidentify-file"; import DeidentifyFileRequest from "../../model/request/deidentify-file"; import DeidentifyFileResponse from "../../model/response/deidentify-file"; import * as fs from 'fs'; -import SkyflowError from "../../../error"; -import SKYFLOW_ERROR_CODE from "../../../error/codes"; +import { SkyflowError } from "skyflow-common"; +import { SKYFLOW_ERROR_CODE } from "skyflow-common"; import GetDetectRunRequest from "../../model/request/get-detect-run"; import Transformations from "../../model/options/deidentify-text/transformations"; import { SkyflowAllError } from "../../types"; diff --git a/src/vault/controller/vault/index.ts b/skyvault/src/vault/controller/vault/index.ts similarity index 99% rename from src/vault/controller/vault/index.ts rename to skyvault/src/vault/controller/vault/index.ts index 14adccb4..9df488c5 100644 --- a/src/vault/controller/vault/index.ts +++ b/skyvault/src/vault/controller/vault/index.ts @@ -22,16 +22,16 @@ import FileUploadResponse from '../../model/response/file-upload'; import TokenizeResponse from '../../model/response/tokenize'; import TokenizeRequest from '../../model/request/tokenize'; import { InsertResponseType, ParsedDetokenizeResponse, ParsedInsertBatchResponse, RecordsResponse, StringKeyValueMapType, TokenizeRequestType } from '../../types'; -import { generateSDKMetrics, getBearerToken, MessageType, parameterizedString, printLog, TYPES, SDK, removeSDKVersion, RedactionType, SKYFLOW, SkyflowRecordError, HTTP_STATUS_CODE, HTTP_HEADER, CONTENT_TYPE, ENCODING_TYPE } from '../../../utils'; +import { generateSDKMetrics, getBearerToken, MessageType, parameterizedString, printLog, TYPES, SDK, removeSDKVersion, RedactionType, SKYFLOW, SkyflowRecordError, HTTP_STATUS_CODE, HTTP_HEADER, CONTENT_TYPE, ENCODING_TYPE } from "skyflow-common"; import GetColumnRequest from '../../model/request/get-column'; -import logs from '../../../utils/logs'; +import { logs } from "skyflow-common"; import VaultClient from '../../client'; import { validateDeleteRequest, validateDetokenizeRequest, validateGetColumnRequest, validateGetRequest, validateInsertRequest, validateQueryRequest, validateTokenizeRequest, validateUpdateRequest, validateUploadFileRequest } from '../../../utils/validations'; import path from 'path'; import { Records } from '../../../ _generated_/rest/api/resources/records/client/Client'; import FileUploadOptions from '../../model/options/fileUpload'; -import SkyflowError from '../../../error'; -import SKYFLOW_ERROR_CODE from '../../../error/codes'; +import { SkyflowError } from "skyflow-common"; +import { SKYFLOW_ERROR_CODE } from "skyflow-common"; class VaultController { diff --git a/src/vault/model/options/deidentify-file/bleep-audio/index.ts b/skyvault/src/vault/model/options/deidentify-file/bleep-audio/index.ts similarity index 100% rename from src/vault/model/options/deidentify-file/bleep-audio/index.ts rename to skyvault/src/vault/model/options/deidentify-file/bleep-audio/index.ts diff --git a/src/vault/model/options/deidentify-file/index.ts b/skyvault/src/vault/model/options/deidentify-file/index.ts similarity index 99% rename from src/vault/model/options/deidentify-file/index.ts rename to skyvault/src/vault/model/options/deidentify-file/index.ts index b4cb0e7a..25c0ecc3 100644 --- a/src/vault/model/options/deidentify-file/index.ts +++ b/skyvault/src/vault/model/options/deidentify-file/index.ts @@ -1,4 +1,4 @@ -import { DetectEntities, DetectOutputTranscription, MaskingMethod } from "../../../../utils"; +import { DetectEntities, DetectOutputTranscription, MaskingMethod } from "skyflow-common"; import TokenFormat from "../deidentify-text/token-format"; import Transformations from "../deidentify-text/transformations"; import { Bleep } from './bleep-audio' diff --git a/src/vault/model/options/deidentify-text/index.ts b/skyvault/src/vault/model/options/deidentify-text/index.ts similarity index 96% rename from src/vault/model/options/deidentify-text/index.ts rename to skyvault/src/vault/model/options/deidentify-text/index.ts index 0fcbb078..10e446e0 100644 --- a/src/vault/model/options/deidentify-text/index.ts +++ b/skyvault/src/vault/model/options/deidentify-text/index.ts @@ -1,4 +1,4 @@ -import { DetectEntities } from "../../../../utils"; +import { DetectEntities } from "skyflow-common"; import TokenFormat from "./token-format"; import Transformations from "./transformations"; diff --git a/src/vault/model/options/deidentify-text/token-format/index.ts b/skyvault/src/vault/model/options/deidentify-text/token-format/index.ts similarity index 94% rename from src/vault/model/options/deidentify-text/token-format/index.ts rename to skyvault/src/vault/model/options/deidentify-text/token-format/index.ts index a2c6fb07..8ee2e7e1 100644 --- a/src/vault/model/options/deidentify-text/token-format/index.ts +++ b/skyvault/src/vault/model/options/deidentify-text/token-format/index.ts @@ -1,4 +1,4 @@ -import { DetectEntities, TokenType } from "../../../../../utils"; +import { DetectEntities, TokenType } from "skyflow-common"; class TokenFormat { // Fields diff --git a/src/vault/model/options/deidentify-text/transformations/index.ts b/skyvault/src/vault/model/options/deidentify-text/transformations/index.ts similarity index 89% rename from src/vault/model/options/deidentify-text/transformations/index.ts rename to skyvault/src/vault/model/options/deidentify-text/transformations/index.ts index 85fa2612..d589c074 100644 --- a/src/vault/model/options/deidentify-text/transformations/index.ts +++ b/skyvault/src/vault/model/options/deidentify-text/transformations/index.ts @@ -1,4 +1,4 @@ -import { DetectEntities } from "../../../../../utils"; +import { DetectEntities } from "skyflow-common"; class Transformations { // Fields diff --git a/src/vault/model/options/detokenize/index.ts b/skyvault/src/vault/model/options/detokenize/index.ts similarity index 82% rename from src/vault/model/options/detokenize/index.ts rename to skyvault/src/vault/model/options/detokenize/index.ts index 3fd1b9f6..e9861481 100644 --- a/src/vault/model/options/detokenize/index.ts +++ b/skyvault/src/vault/model/options/detokenize/index.ts @@ -1,13 +1,14 @@ -import { LogLevel, MessageType, printLog } from '../../../../utils'; -import logs from '../../../../utils/logs'; +import { LogLevel, MessageType, printLog } from "skyflow-common"; +import { logs } from "skyflow-common"; +import { BaseOptions } from "skyflow-common"; -class DetokenizeOptions { +class DetokenizeOptions extends BaseOptions { // Fields with default values private continueOnError?: boolean; private downloadUrl?: boolean; // Constructor - constructor() { } + constructor() { super(); } // Setters setContinueOnError(continueOnError: boolean) { diff --git a/src/vault/model/options/fileUpload/index.ts b/skyvault/src/vault/model/options/fileUpload/index.ts similarity index 89% rename from src/vault/model/options/fileUpload/index.ts rename to skyvault/src/vault/model/options/fileUpload/index.ts index bd44772f..8a4dc9b2 100644 --- a/src/vault/model/options/fileUpload/index.ts +++ b/skyvault/src/vault/model/options/fileUpload/index.ts @@ -1,5 +1,6 @@ +import { BaseOptions } from "skyflow-common"; -class FileUploadOptions { +class FileUploadOptions extends BaseOptions { // Fields with default values private filePath?: string; private base64?: string; @@ -8,7 +9,7 @@ class FileUploadOptions { private skyflowId?: string; // Constructor - constructor() { } + constructor() { super(); } // Setters setFilePath(filePath: string): void { diff --git a/src/vault/model/options/get/index.ts b/skyvault/src/vault/model/options/get/index.ts similarity index 93% rename from src/vault/model/options/get/index.ts rename to skyvault/src/vault/model/options/get/index.ts index a59f3464..ee607aa1 100644 --- a/src/vault/model/options/get/index.ts +++ b/skyvault/src/vault/model/options/get/index.ts @@ -1,8 +1,9 @@ // Imports -import { LogLevel, MessageType, OrderByEnum, printLog, RedactionType } from "../../../../utils"; -import logs from '../../../../utils/logs'; +import { LogLevel, MessageType, OrderByEnum, printLog, RedactionType } from "skyflow-common"; +import { logs } from "skyflow-common"; +import { BaseOptions } from "skyflow-common"; -class GetOptions { +class GetOptions extends BaseOptions { // Fields private redactionType?: RedactionType; private returnTokens?: boolean; @@ -15,7 +16,7 @@ class GetOptions { private orderBy?: OrderByEnum; // Constructor - constructor() {} + constructor() { super(); } // Setters setRedactionType(redactionType: RedactionType) { diff --git a/src/vault/model/options/insert/index.ts b/skyvault/src/vault/model/options/insert/index.ts similarity index 89% rename from src/vault/model/options/insert/index.ts rename to skyvault/src/vault/model/options/insert/index.ts index 9d0dfdcd..9a32be2d 100644 --- a/src/vault/model/options/insert/index.ts +++ b/skyvault/src/vault/model/options/insert/index.ts @@ -1,7 +1,8 @@ -import { TokenMode } from "../../../../utils"; +import { TokenMode } from "skyflow-common"; +import { BaseOptions } from "skyflow-common"; //imports -class InsertOptions { +class InsertOptions extends BaseOptions { // Fields private returnTokens?: boolean; private upsert?: string; @@ -11,7 +12,7 @@ class InsertOptions { private continueOnError?: boolean; // Constructor - constructor() { } + constructor() { super(); } // Setters setReturnTokens(returnTokens: boolean) { diff --git a/src/vault/model/options/reidentify-text/index.ts b/skyvault/src/vault/model/options/reidentify-text/index.ts similarity index 94% rename from src/vault/model/options/reidentify-text/index.ts rename to skyvault/src/vault/model/options/reidentify-text/index.ts index 3cbd69e9..78659816 100644 --- a/src/vault/model/options/reidentify-text/index.ts +++ b/skyvault/src/vault/model/options/reidentify-text/index.ts @@ -1,4 +1,4 @@ -import { DetectEntities } from "../../../../utils"; +import { DetectEntities } from "skyflow-common"; class ReidentifyTextOptions { // Fields diff --git a/src/vault/model/options/update/index.ts b/skyvault/src/vault/model/options/update/index.ts similarity index 83% rename from src/vault/model/options/update/index.ts rename to skyvault/src/vault/model/options/update/index.ts index f6dec32a..e86812b9 100644 --- a/src/vault/model/options/update/index.ts +++ b/skyvault/src/vault/model/options/update/index.ts @@ -1,7 +1,8 @@ //imports -import { TokenMode } from "../../../../utils"; +import { TokenMode } from "skyflow-common"; +import { BaseOptions } from "skyflow-common"; -class UpdateOptions { +class UpdateOptions extends BaseOptions { //fields private returnTokens?: boolean; @@ -10,6 +11,7 @@ class UpdateOptions { // Constructor constructor() { + super(); } setReturnTokens(returnTokens: boolean) { diff --git a/src/vault/model/request/deidentify-file/index.ts b/skyvault/src/vault/model/request/deidentify-file/index.ts similarity index 100% rename from src/vault/model/request/deidentify-file/index.ts rename to skyvault/src/vault/model/request/deidentify-file/index.ts diff --git a/src/vault/model/request/deidentify-text/index.ts b/skyvault/src/vault/model/request/deidentify-text/index.ts similarity index 100% rename from src/vault/model/request/deidentify-text/index.ts rename to skyvault/src/vault/model/request/deidentify-text/index.ts diff --git a/src/vault/model/request/delete/index.ts b/skyvault/src/vault/model/request/delete/index.ts similarity index 84% rename from src/vault/model/request/delete/index.ts rename to skyvault/src/vault/model/request/delete/index.ts index f3980922..3bcccee2 100644 --- a/src/vault/model/request/delete/index.ts +++ b/skyvault/src/vault/model/request/delete/index.ts @@ -1,12 +1,14 @@ //imports +import { BaseDeleteRequest } from "skyflow-common"; -class DeleteRequest { +class DeleteRequest extends BaseDeleteRequest { //fields private _table: string; private _ids: Array; // Constructor constructor(table: string, deleteIds: Array) { + super(); this._table = table; this._ids = deleteIds; } diff --git a/src/vault/model/request/detokenize/index.ts b/skyvault/src/vault/model/request/detokenize/index.ts similarity index 77% rename from src/vault/model/request/detokenize/index.ts rename to skyvault/src/vault/model/request/detokenize/index.ts index 01a17623..66f08dca 100644 --- a/src/vault/model/request/detokenize/index.ts +++ b/skyvault/src/vault/model/request/detokenize/index.ts @@ -1,13 +1,15 @@ //imports import { DetokenizeData } from "../../../types"; +import { BaseDetokenizeRequest } from "skyflow-common"; -class DetokenizeRequest { +class DetokenizeRequest extends BaseDetokenizeRequest { //fields private _data: DetokenizeData[]; // Constructor constructor(data: DetokenizeData[]) { + super(); this._data = data; } diff --git a/src/vault/model/request/file-upload/index.ts b/skyvault/src/vault/model/request/file-upload/index.ts similarity index 87% rename from src/vault/model/request/file-upload/index.ts rename to skyvault/src/vault/model/request/file-upload/index.ts index 2ef75143..ef9a94a3 100644 --- a/src/vault/model/request/file-upload/index.ts +++ b/skyvault/src/vault/model/request/file-upload/index.ts @@ -1,14 +1,16 @@ // Imports -import { LogLevel, MessageType, printLog } from '../../../../utils'; -import logs from '../../../../utils/logs'; +import { LogLevel, MessageType, printLog } from "skyflow-common"; +import { logs } from "skyflow-common"; +import { BaseFileUploadRequest } from "skyflow-common"; -class FileUploadRequest { +class FileUploadRequest extends BaseFileUploadRequest { private _table: string; private _columnName: string; private _legacySkyflowId?: string; // Constructor constructor(table: string, columnNameOrSkyflowId: string, columnName?: string) { + super(); this._table = table; if (columnName !== undefined) { diff --git a/src/vault/model/request/get-column/index.ts b/skyvault/src/vault/model/request/get-column/index.ts similarity index 100% rename from src/vault/model/request/get-column/index.ts rename to skyvault/src/vault/model/request/get-column/index.ts diff --git a/src/vault/model/request/get-detect-run/index.ts b/skyvault/src/vault/model/request/get-detect-run/index.ts similarity index 100% rename from src/vault/model/request/get-detect-run/index.ts rename to skyvault/src/vault/model/request/get-detect-run/index.ts diff --git a/src/vault/model/request/get/index.ts b/skyvault/src/vault/model/request/get/index.ts similarity index 80% rename from src/vault/model/request/get/index.ts rename to skyvault/src/vault/model/request/get/index.ts index 575c2009..d6900c64 100644 --- a/src/vault/model/request/get/index.ts +++ b/skyvault/src/vault/model/request/get/index.ts @@ -1,8 +1,9 @@ //imports -import { RedactionType } from "../../../../utils"; +import { RedactionType } from "skyflow-common"; +import { BaseGetRequest } from "skyflow-common"; -class GetRequest { +class GetRequest extends BaseGetRequest { //fields private _table: string; @@ -10,6 +11,7 @@ class GetRequest { // Constructor constructor(table: string, _ids: Array) { + super(); this._table = table; this._ids = _ids; } diff --git a/src/vault/model/request/inkove/index.ts b/skyvault/src/vault/model/request/inkove/index.ts similarity index 77% rename from src/vault/model/request/inkove/index.ts rename to skyvault/src/vault/model/request/inkove/index.ts index d063bb9d..58c6a263 100644 --- a/src/vault/model/request/inkove/index.ts +++ b/skyvault/src/vault/model/request/inkove/index.ts @@ -1,8 +1,9 @@ //imports -import { RequestMethod } from "../../../../utils"; +import { RequestMethod } from "skyflow-common"; import { StringKeyValueMapType } from "../../../types"; +import { BaseInvokeConnectionRequest } from "skyflow-common"; -class InvokeConnectionRequest { +class InvokeConnectionRequest extends BaseInvokeConnectionRequest { //fields method: RequestMethod; queryParams?: StringKeyValueMapType; @@ -11,6 +12,7 @@ class InvokeConnectionRequest { headers?: StringKeyValueMapType; constructor(method: RequestMethod, body?: StringKeyValueMapType, headers?: StringKeyValueMapType, pathParams?: StringKeyValueMapType, queryParams?: StringKeyValueMapType) { + super(); this.method = method; this.pathParams = pathParams; this.queryParams = queryParams; diff --git a/src/vault/model/request/insert/index.ts b/skyvault/src/vault/model/request/insert/index.ts similarity index 85% rename from src/vault/model/request/insert/index.ts rename to skyvault/src/vault/model/request/insert/index.ts index ee251c9f..6ed61df4 100644 --- a/src/vault/model/request/insert/index.ts +++ b/skyvault/src/vault/model/request/insert/index.ts @@ -1,6 +1,7 @@ //imports +import { BaseInsertRequest } from "skyflow-common"; -class InsertRequest { +class InsertRequest extends BaseInsertRequest { //fields private _table: string; @@ -8,6 +9,7 @@ class InsertRequest { // Constructor constructor(table: string, data: Record[]) { + super(); this._table = table; this._data = data; } diff --git a/src/vault/model/request/query/index.ts b/skyvault/src/vault/model/request/query/index.ts similarity index 73% rename from src/vault/model/request/query/index.ts rename to skyvault/src/vault/model/request/query/index.ts index 1fe3da91..e0808671 100644 --- a/src/vault/model/request/query/index.ts +++ b/skyvault/src/vault/model/request/query/index.ts @@ -1,8 +1,10 @@ +import { BaseQueryRequest } from "skyflow-common"; -class QueryRequest { +class QueryRequest extends BaseQueryRequest { private _query: string; constructor(query: string) { + super(); this._query = query; } diff --git a/src/vault/model/request/reidentify-text/index.ts b/skyvault/src/vault/model/request/reidentify-text/index.ts similarity index 100% rename from src/vault/model/request/reidentify-text/index.ts rename to skyvault/src/vault/model/request/reidentify-text/index.ts diff --git a/src/vault/model/request/tokenize/index.ts b/skyvault/src/vault/model/request/tokenize/index.ts similarity index 80% rename from src/vault/model/request/tokenize/index.ts rename to skyvault/src/vault/model/request/tokenize/index.ts index f7ef067d..3fe6bf03 100644 --- a/src/vault/model/request/tokenize/index.ts +++ b/skyvault/src/vault/model/request/tokenize/index.ts @@ -1,14 +1,16 @@ //imports import { TokenizeRequestType } from "../../../types"; +import { BaseTokenizeRequest } from "skyflow-common"; -class TokenizeRequest { +class TokenizeRequest extends BaseTokenizeRequest { //fields private _values: Array; // Constructor constructor(values: Array) { + super(); this._values = values; } diff --git a/src/vault/model/request/update/index.ts b/skyvault/src/vault/model/request/update/index.ts similarity index 85% rename from src/vault/model/request/update/index.ts rename to skyvault/src/vault/model/request/update/index.ts index a749f74c..885e6910 100644 --- a/src/vault/model/request/update/index.ts +++ b/skyvault/src/vault/model/request/update/index.ts @@ -1,6 +1,7 @@ //imports +import { BaseUpdateRequest } from "skyflow-common"; -class UpdateRequest { +class UpdateRequest extends BaseUpdateRequest { //fields private _table: string; @@ -8,6 +9,7 @@ class UpdateRequest { // Constructor constructor(table: string, data: Record) { + super(); this._table = table; this._data = data; } diff --git a/src/vault/model/response/deidentify-file/index.ts b/skyvault/src/vault/model/response/deidentify-file/index.ts similarity index 96% rename from src/vault/model/response/deidentify-file/index.ts rename to skyvault/src/vault/model/response/deidentify-file/index.ts index b39af6d8..9732c40a 100644 --- a/src/vault/model/response/deidentify-file/index.ts +++ b/skyvault/src/vault/model/response/deidentify-file/index.ts @@ -1,4 +1,4 @@ -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; class DeidentifyFileResponse { // fields diff --git a/src/vault/model/response/deidentify-text/index.ts b/skyvault/src/vault/model/response/deidentify-text/index.ts similarity index 95% rename from src/vault/model/response/deidentify-text/index.ts rename to skyvault/src/vault/model/response/deidentify-text/index.ts index 8beca5d9..fb78984f 100644 --- a/src/vault/model/response/deidentify-text/index.ts +++ b/skyvault/src/vault/model/response/deidentify-text/index.ts @@ -1,7 +1,7 @@ //imports import { IndexRange } from "../../../types"; -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; class DeidentifyTextResponse { //fields diff --git a/src/vault/model/response/delete/index.ts b/skyvault/src/vault/model/response/delete/index.ts similarity index 73% rename from src/vault/model/response/delete/index.ts rename to skyvault/src/vault/model/response/delete/index.ts index 737176bb..d9fe8983 100644 --- a/src/vault/model/response/delete/index.ts +++ b/skyvault/src/vault/model/response/delete/index.ts @@ -1,8 +1,9 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; +import { BaseDeleteResponse } from "skyflow-common"; -class DeleteResponse { +class DeleteResponse extends BaseDeleteResponse { //fields @@ -14,6 +15,7 @@ class DeleteResponse { * @deprecated Passing undefined for deletedIds is no longer supported. Pass empty array [] instead. */ constructor({ deletedIds, errors }: { deletedIds: Array, errors: Array | null}) { + super(); this.deletedIds = deletedIds; this.errors = errors; } diff --git a/src/vault/model/response/detokenize/index.ts b/skyvault/src/vault/model/response/detokenize/index.ts similarity index 74% rename from src/vault/model/response/detokenize/index.ts rename to skyvault/src/vault/model/response/detokenize/index.ts index 99b94710..7601b389 100644 --- a/src/vault/model/response/detokenize/index.ts +++ b/skyvault/src/vault/model/response/detokenize/index.ts @@ -1,9 +1,10 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; import { ErrorDetokenizeResponse, SuccessDetokenizeResponse } from "../../../types"; +import { BaseDetokenizeResponse } from "skyflow-common"; -class DetokenizeResponse { +class DetokenizeResponse extends BaseDetokenizeResponse { //fields detokenizedFields: Array | null; @@ -11,6 +12,7 @@ class DetokenizeResponse { errors: Array | null; constructor({ detokenizedFields, errors }: { detokenizedFields: Array | null, errors: Array | null }) { + super(); this.detokenizedFields = detokenizedFields; this.errors = errors; } diff --git a/src/vault/model/response/file-upload/index.ts b/skyvault/src/vault/model/response/file-upload/index.ts similarity index 64% rename from src/vault/model/response/file-upload/index.ts rename to skyvault/src/vault/model/response/file-upload/index.ts index ce0ef2c8..9b778131 100644 --- a/src/vault/model/response/file-upload/index.ts +++ b/skyvault/src/vault/model/response/file-upload/index.ts @@ -1,8 +1,9 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; +import { BaseFileUploadResponse } from "skyflow-common"; -class FileUploadResponse { +class FileUploadResponse extends BaseFileUploadResponse { //fields skyflowId: string; @@ -10,6 +11,7 @@ class FileUploadResponse { errors: Array | null; constructor({ skyflowId, errors }: { skyflowId: string, errors: Array | null }) { + super(); this.skyflowId = skyflowId; this.errors = errors; } diff --git a/src/vault/model/response/get/index.ts b/skyvault/src/vault/model/response/get/index.ts similarity index 70% rename from src/vault/model/response/get/index.ts rename to skyvault/src/vault/model/response/get/index.ts index 3a7c465f..6dcb4256 100644 --- a/src/vault/model/response/get/index.ts +++ b/skyvault/src/vault/model/response/get/index.ts @@ -1,9 +1,10 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; import { GetResponseData } from "../../../types"; +import { BaseGetResponse } from "skyflow-common"; -class GetResponse { +class GetResponse extends BaseGetResponse { //fields data: Array; @@ -11,6 +12,7 @@ class GetResponse { errors: Array | null; constructor({ data, errors }: { data: Array, errors: Array | null}) { + super(); this.data = data; this.errors = errors; } diff --git a/src/vault/model/response/insert/index.ts b/skyvault/src/vault/model/response/insert/index.ts similarity index 70% rename from src/vault/model/response/insert/index.ts rename to skyvault/src/vault/model/response/insert/index.ts index c50dafdf..b55db684 100644 --- a/src/vault/model/response/insert/index.ts +++ b/skyvault/src/vault/model/response/insert/index.ts @@ -1,8 +1,9 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; import { InsertResponseType } from "../../../types"; +import { BaseInsertResponse } from "skyflow-common"; -class InsertResponse { +class InsertResponse extends BaseInsertResponse { //fields insertedFields: Array; @@ -13,7 +14,8 @@ class InsertResponse { * @deprecated Passing null for insertedFields is no longer supported. Pass empty array [] instead. */ constructor({ insertedFields, errors }: { insertedFields: Array, errors: Array | null }) { - this.insertedFields = insertedFields; + super(); + this.insertedFields = insertedFields; this.errors = errors; } diff --git a/src/vault/model/response/invoke/invoke.ts b/skyvault/src/vault/model/response/invoke/invoke.ts similarity index 68% rename from src/vault/model/response/invoke/invoke.ts rename to skyvault/src/vault/model/response/invoke/invoke.ts index 617bb708..85cf9f94 100644 --- a/src/vault/model/response/invoke/invoke.ts +++ b/skyvault/src/vault/model/response/invoke/invoke.ts @@ -1,8 +1,9 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; +import { BaseInvokeConnectionResponse } from "skyflow-common"; -class InvokeConnectionResponse { +class InvokeConnectionResponse extends BaseInvokeConnectionResponse { //fields data?: Object; @@ -11,6 +12,7 @@ class InvokeConnectionResponse { errors: Array | null; constructor({ data, metadata, errors }: { data?: object, metadata?: Record, errors: Array | null }) { + super(); this.data = data; this.metadata = metadata; this.errors = errors; diff --git a/src/vault/model/response/query/index.ts b/skyvault/src/vault/model/response/query/index.ts similarity index 70% rename from src/vault/model/response/query/index.ts rename to skyvault/src/vault/model/response/query/index.ts index a67321ec..b007f629 100644 --- a/src/vault/model/response/query/index.ts +++ b/skyvault/src/vault/model/response/query/index.ts @@ -1,9 +1,10 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; import { QueryResponseType } from "../../../types"; +import { BaseQueryResponse } from "skyflow-common"; -class QueryResponse { +class QueryResponse extends BaseQueryResponse { //fields fields: Array; @@ -11,6 +12,7 @@ class QueryResponse { errors: Array | null; constructor( { fields, errors }: { fields: Array, errors: Array | null }) { + super(); this.fields = fields; this.errors = errors; } diff --git a/src/vault/model/response/reidentify-text/index.ts b/skyvault/src/vault/model/response/reidentify-text/index.ts similarity index 100% rename from src/vault/model/response/reidentify-text/index.ts rename to skyvault/src/vault/model/response/reidentify-text/index.ts diff --git a/src/vault/model/response/tokenize/index.ts b/skyvault/src/vault/model/response/tokenize/index.ts similarity index 65% rename from src/vault/model/response/tokenize/index.ts rename to skyvault/src/vault/model/response/tokenize/index.ts index ea279e47..56b6ea2d 100644 --- a/src/vault/model/response/tokenize/index.ts +++ b/skyvault/src/vault/model/response/tokenize/index.ts @@ -1,8 +1,9 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; +import { BaseTokenizeResponse } from "skyflow-common"; -class TokenizeResponse { +class TokenizeResponse extends BaseTokenizeResponse { //fields tokens: Array; @@ -10,6 +11,7 @@ class TokenizeResponse { errors: Array | null; constructor({ tokens, errors }: { tokens: Array, errors: Array | null }) { + super(); this.tokens = tokens; this.errors = errors; } diff --git a/src/vault/model/response/update/index.ts b/skyvault/src/vault/model/response/update/index.ts similarity index 64% rename from src/vault/model/response/update/index.ts rename to skyvault/src/vault/model/response/update/index.ts index 6696dffe..9f215ff8 100644 --- a/src/vault/model/response/update/index.ts +++ b/skyvault/src/vault/model/response/update/index.ts @@ -1,9 +1,10 @@ //imports -import { SkyflowRecordError } from "../../../../utils"; +import { SkyflowRecordError } from "skyflow-common"; import { InsertResponseType } from "../../../types"; +import { BaseUpdateResponse } from "skyflow-common"; -class UpdateResponse { +class UpdateResponse extends BaseUpdateResponse { //fields updatedField: InsertResponseType; @@ -11,7 +12,8 @@ class UpdateResponse { errors: Array | null; constructor({ updatedField, errors }: { updatedField: InsertResponseType, errors: Array | null }) { - this.updatedField = updatedField; + super(); + this.updatedField = updatedField; this.errors = errors; } diff --git a/skyvault/src/vault/skyflow/index.ts b/skyvault/src/vault/skyflow/index.ts new file mode 100644 index 00000000..aaa0f94f --- /dev/null +++ b/skyvault/src/vault/skyflow/index.ts @@ -0,0 +1,201 @@ +import { + BaseSkyflow, + BaseVaultEntry, + CONFIG, + Env, + getVaultURL, + LogLevel, + MessageType, + parameterizedString, + printLog, + logs, + SkyflowError, + SKYFLOW_ERROR_CODE, + Credentials, + VaultConfig, + ConnectionConfig, + validateConnectionConfig, + validateUpdateConnectionConfig, + setSdkIdentity, +} from "skyflow-common"; +import { SkyflowConfig } from "../types"; +import VaultController from "../controller/vault"; +import ConnectionController from "../controller/connections"; +import VaultClient from "../client"; +import DetectController from "../controller/detect"; +import { SDK_NAME, SDK_VERSION } from "../../version"; + +// Stamp the SDK identity so error text + telemetry self-report this product. +setSdkIdentity({ name: SDK_NAME, version: SDK_VERSION }); + +/** A configured vault: the client plus the controllers skyvault exposes. */ +interface VaultEntry extends BaseVaultEntry { + vaultController: VaultController; + detectController: DetectController; +} + +/** A configured connection: the client plus its connection controller. */ +interface ConnectionEntry { + config: ConnectionConfig; + client: VaultClient; + connectionController: ConnectionController; +} + +/** + * Skyflow is the SkyVault (`skyflow-node`) SDK facade. The vault-config / + * credential / log-level management is inherited from the shared `BaseSkyflow` + * in `common`; this layer adds the SkyVault-only surfaces: the detect + * controller on each vault, and the connection registry + `connection()`. + */ +class Skyflow extends BaseSkyflow { + private connectionClients: { [connectionId: string]: ConnectionEntry } = {}; + + constructor(config: SkyflowConfig) { + super(); + // Common vault + credential + log-level setup (validates config, logs, + // registers vaultConfigs via buildVaultEntry). + this.initializeSkyflow(config); + + printLog(logs.infoLogs.VALIDATING_CONNECTION_CONFIG, MessageType.LOG, this.logLevel); + config.connectionConfigs?.map((connectionConfig) => { + this.addConnectionConfig(connectionConfig); + }); + printLog(logs.infoLogs.CLIENT_INITIALIZED, MessageType.LOG, this.logLevel); + } + + // --- layer-specific vault entry (vault + detect controllers) --------------- + protected buildVaultEntry(config: VaultConfig): VaultEntry { + const env = config.env || Env.PROD; + const vaultUrl = getVaultURL(config.clusterId, env); + const client = new VaultClient( + vaultUrl, + config.vaultId, + config?.credentials, + this.commonCredentials, + this.logLevel, + ); + const vaultController = new VaultController(client); + const detectController = new DetectController(client); + return { config, client, vaultController, detectController }; + } + + // --- log level / credentials also propagate to connection clients ---------- + protected applyLogLevel(): void { + super.applyLogLevel(); + Object.values(this.connectionClients).forEach((entry) => entry.client.setLogLevel(this.logLevel)); + } + + protected applyCredentials(): void { + super.applyCredentials(); + Object.values(this.connectionClients).forEach((entry) => + entry.client.updateSkyflowCredentials(this.commonCredentials), + ); + } + + // --- connections (SkyVault only) ------------------------------------------- + private addConnectionClient(config: ConnectionConfig): void { + const client = new VaultClient( + config.connectionUrl, + "", + config?.credentials, + this.commonCredentials, + this.logLevel, + ); + const connectionController = new ConnectionController(client); + printLog( + parameterizedString(logs.infoLogs.CONNECTION_CONTROLLER_INITIALIZED, [config.connectionId]), + MessageType.LOG, + this.logLevel, + ); + this.connectionClients[config.connectionId] = { config, client, connectionController }; + } + + addConnectionConfig(config: ConnectionConfig): void { + validateConnectionConfig(config, this.logLevel); + this.throwErrorIfConnectionIdExists(config?.connectionId); + this.addConnectionClient(config); + } + + updateConnectionConfig(config: ConnectionConfig): void { + validateUpdateConnectionConfig(config, this.logLevel); + const existing = this.connectionClients[config.connectionId]; + if (!existing) { + this.throwErrorForUnknownConnectionId(config.connectionId); + return; + } + const updatedConfig = { ...existing.config, ...config }; + existing.config = updatedConfig; + existing.client.updateClientConfig( + updatedConfig.connectionUrl, + "", + updatedConfig.credentials, + this.commonCredentials, + this.logLevel, + ); + } + + getConnectionConfig(connectionId: string): ConnectionConfig { + if (!connectionId) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_ID_VALIDATION); + } + const existing = this.connectionClients[connectionId]; + if (!existing) { + this.throwErrorForUnknownConnectionId(connectionId); + } + return existing.config; + } + + removeConnectionConfig(connectionId: string): void { + if (!this.connectionClients[connectionId]) { + this.throwErrorForUnknownConnectionId(connectionId); + return; + } + delete this.connectionClients[connectionId]; + } + + // --- controller accessors -------------------------------------------------- + vault(vaultId?: string): VaultController { + return this.resolveVaultEntry(vaultId).vaultController; + } + + detect(vaultId?: string): DetectController { + return this.resolveVaultEntry(vaultId).detectController; + } + + connection(connectionId?: string): ConnectionController { + if (Object.keys(this.connectionClients).length === 0) { + throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_CLIENTS); + } + const clientId = connectionId || Object.keys(this.connectionClients)[0]; + const entry = this.connectionClients[clientId]; + if (entry?.connectionController) { + return entry.connectionController; + } + this.throwErrorForUnknownConnectionId(clientId); + // Unreachable: throwErrorForUnknownConnectionId always throws. + throw new SkyflowError(SKYFLOW_ERROR_CODE.INTERNAL_SERVER_ERROR); + } + + // --- connection error helpers ---------------------------------------------- + private throwErrorIfConnectionIdExists(id: string): void { + if (Object.keys(this.connectionClients).includes(id)) { + printLog( + parameterizedString(logs.infoLogs[`${CONFIG.CONNECTION_ID}_CONFIG_EXISTS`], [id]), + MessageType.ERROR, + this.logLevel, + ); + throw new SkyflowError(SKYFLOW_ERROR_CODE.CONNECTION_ID_EXITS_IN_CONFIG_LIST, [id]); + } + } + + private throwErrorForUnknownConnectionId(id: string): void { + printLog( + parameterizedString(logs.infoLogs[`${CONFIG.CONNECTION_ID}_CONFIG_DOES_NOT_EXIST`], [id]), + MessageType.ERROR, + this.logLevel, + ); + throw new SkyflowError(SKYFLOW_ERROR_CODE.CONNECTION_ID_NOT_IN_CONFIG_LIST, [id]); + } +} + +export default Skyflow; diff --git a/src/vault/types/index.ts b/skyvault/src/vault/types/index.ts similarity index 72% rename from src/vault/types/index.ts rename to skyvault/src/vault/types/index.ts index 7821f1cd..5e78002e 100644 --- a/src/vault/types/index.ts +++ b/skyvault/src/vault/types/index.ts @@ -1,31 +1,28 @@ -import { LogLevel, RedactionType, SkyflowRecordError } from "../../utils"; -import ConnectionConfig from "../config/connection"; -import VaultConfig from "../config/vault" -import Credentials from "../config/credentials"; +import { RedactionType, SkyflowRecordError } from "skyflow-common"; +import { ConnectionConfig } from "skyflow-common"; +import { VaultConfig } from "skyflow-common"; import VaultController from "../controller/vault"; import ConnectionController from "../controller/connections"; import VaultClient from "../client"; import DetectController from "../controller/detect"; -import SkyflowError from "../../error"; - -export interface SkyflowConfig { - vaultConfigs?: VaultConfig[]; - connectionConfigs?: ConnectionConfig[]; - skyflowCredentials?: Credentials; - logLevel?: LogLevel; -} - -export type FileInput = - | Filepath - | FileObject - -export interface Filepath { - filePath: string; -} - -export interface FileObject { - file: File; -} +import { SkyflowError } from "skyflow-common"; +import { InsertResponseType } from "skyflow-common"; + +// Re-export the contract-agnostic types that now live in skyflow-common, so the +// public API surface of skyflow-node continues to expose them from here. +export type { + SkyflowConfig, + FileInput, + Filepath, + FileObject, + InsertResponseType, + GetResponseData, + QueryResponseType, + StringKeyValueMapType, + TokenizeRequestType, + IndexRange, + ServiceAccountResponseError, +} from "skyflow-common"; export interface ClientConfig { config: VaultConfig | ConnectionConfig; @@ -39,29 +36,6 @@ export interface ClientObj { [vaultId: string]: ClientConfig; } -export interface InsertResponseType { - skyflowId: string; - /** @deprecated Renamed to skyflowId. Will be removed in v3. */ - skyflow_id?: string; - [key: string]: unknown; -} - -export interface GetResponseData { - [key: string]: unknown; -} - -export interface QueryResponseType { - [key: string]: unknown; -} - -export interface StringKeyValueMapType { - [key: string]: object | string; -} - -export interface TokenizeRequestType { - columnGroup: string; - value: string; -} export interface SuccessDetokenizeResponse { token: string; value: string; @@ -142,20 +116,6 @@ export type SkyflowErrorData = | SkyflowApiErrorLegacyBody['error'] | undefined; -export type ServiceAccountResponseError = { - rawResponse?: { - headers?: { - get: (header: string) => string | undefined; - }; - }; - body: any; - response?: { - status?: number | string; - }; - message: string; - [key: string]: any; -}; - export interface RecordsResponse> { records: T[]; requestId: string; @@ -175,14 +135,10 @@ export interface SkyflowIdResponse { /** @deprecated Renamed to skyflowId. Will be removed in v3. */ skyflow_id?: string; } - + export interface TokensResponse extends SkyflowIdResponse { tokens?: Record; } -export interface IndexRange { - start?: number; - end?: number; -} export type DeidentifyFileOutputProcessedFileType = | "entities" diff --git a/skyvault/src/version.ts b/skyvault/src/version.ts new file mode 100644 index 00000000..d07c66bc --- /dev/null +++ b/skyvault/src/version.ts @@ -0,0 +1,4 @@ +// SDK version constant for skyflow-node (skyvault). +// Kept in sync with skyvault/package.json "version" by scripts/bump_version.sh. +export const SDK_NAME = "skyflow-node"; +export const SDK_VERSION = "2.1.2"; diff --git a/skyvault/test/demo-credentials/empty.json b/skyvault/test/demo-credentials/empty.json new file mode 100644 index 00000000..e69de29b diff --git a/skyvault/test/demo-credentials/invalidJson.json b/skyvault/test/demo-credentials/invalidJson.json new file mode 100644 index 00000000..c11a9d0a --- /dev/null +++ b/skyvault/test/demo-credentials/invalidJson.json @@ -0,0 +1 @@ +"some test creds" diff --git a/skyvault/test/demo-credentials/noClientId.json b/skyvault/test/demo-credentials/noClientId.json new file mode 100644 index 00000000..10699c21 --- /dev/null +++ b/skyvault/test/demo-credentials/noClientId.json @@ -0,0 +1,3 @@ +{ + "privateKey": "Some private key" +} \ No newline at end of file diff --git a/skyvault/test/demo-credentials/noKeyId.json b/skyvault/test/demo-credentials/noKeyId.json new file mode 100644 index 00000000..9b93da90 --- /dev/null +++ b/skyvault/test/demo-credentials/noKeyId.json @@ -0,0 +1,4 @@ +{ + "privateKey": "Some private key", + "clientID": "Some client ID" +} \ No newline at end of file diff --git a/skyvault/test/demo-credentials/noPrivateKey.json b/skyvault/test/demo-credentials/noPrivateKey.json new file mode 100644 index 00000000..94efc8d8 --- /dev/null +++ b/skyvault/test/demo-credentials/noPrivateKey.json @@ -0,0 +1 @@ +{"data": "this has no data","clientID": "Some client ID", "keyID": "Some key ID","tokenURI":"uri"} \ No newline at end of file diff --git a/skyvault/test/demo-credentials/noTokenURI.json b/skyvault/test/demo-credentials/noTokenURI.json new file mode 100644 index 00000000..f7d86552 --- /dev/null +++ b/skyvault/test/demo-credentials/noTokenURI.json @@ -0,0 +1,5 @@ +{ + "privateKey": "Some private key", + "clientID": "Some client ID", + "keyID": "Some key ID" +} \ No newline at end of file diff --git a/skyvault/test/demo-credentials/valid.json b/skyvault/test/demo-credentials/valid.json new file mode 100644 index 00000000..ac7f5c00 --- /dev/null +++ b/skyvault/test/demo-credentials/valid.json @@ -0,0 +1,7 @@ +{ + "clientID": "test-client-id", + "keyID": "test-key-id", + "tokenURI": "https://manage.skyflow.dev", + "privateKey": "KEY", + "data": "DATA" +} \ No newline at end of file diff --git a/test/mockData/pii.pdf b/skyvault/test/mockData/pii.pdf similarity index 100% rename from test/mockData/pii.pdf rename to skyvault/test/mockData/pii.pdf diff --git a/test/utils/validations.test.js b/skyvault/test/utils/validations.test.js similarity index 98% rename from test/utils/validations.test.js rename to skyvault/test/utils/validations.test.js index 04e108ef..de25251d 100644 --- a/test/utils/validations.test.js +++ b/skyvault/test/utils/validations.test.js @@ -1,13 +1,19 @@ const fs = require('fs'); const path = require('path'); -const { validateDeidentifyFileRequest, validateDeidentifyFileOptions, validateSkyflowConfig, validateVaultConfig, validateUpdateVaultConfig, validateSkyflowCredentials, validateConnectionConfig, validateUpdateConnectionConfig, validateInsertRequest, validateUpdateRequest, validateGetRequest, validateDetokenizeRequest, validateTokenizeRequest, validateDeleteRequest, validateUploadFileRequest, validateQueryRequest, validateDeIdentifyTextRequest, validateReidentifyTextRequest, validateGetDetectRunRequest, validateInvokeConnectionRequest, validateUpdateOptions, validateGetColumnRequest, validateCredentialsWithId, isEnv, isRedactionType, isByot, isOrderBy, isMethod, isLogLevel, isValidAPIKey, validateInsertOptions, validateGetOptions, validateDetokenizeOptions } = require('../../src/utils/validations'); +const { validateDeidentifyFileRequest, validateDeidentifyFileOptions, validateInsertRequest, validateUpdateRequest, validateGetRequest, validateDetokenizeRequest, validateTokenizeRequest, validateDeleteRequest, validateUploadFileRequest, validateQueryRequest, validateDeIdentifyTextRequest, validateReidentifyTextRequest, validateGetDetectRunRequest, validateInvokeConnectionRequest, validateUpdateOptions, validateGetColumnRequest, isRedactionType, isByot, isOrderBy, isMethod, validateInsertOptions, validateGetOptions, validateDetokenizeOptions } = require('../../src/utils/validations'); +// Generic (contract-agnostic) validators now live in skyflow-common. +const { validateSkyflowConfig, validateVaultConfig, validateUpdateVaultConfig, validateSkyflowCredentials, validateConnectionConfig, validateUpdateConnectionConfig, validateCredentialsWithId, isEnv, isLogLevel, isValidAPIKey } = require('skyflow-common'); const DeidentifyFileRequest = require('../../src/vault/model/request/deidentify-file').default; const DeidentifyFileOptions = require('../../src/vault/model/options/deidentify-file').default; -const SKYFLOW_ERROR_CODE = require('../../src/error/codes'); +// NOTE: On main, `require('../../src/error/codes')` returned the ES module +// wrapper (the codes live on `.default`), so `SKYFLOW_ERROR_CODE.` was +// `undefined` and `toThrow(undefined)` only asserted "throws something". We +// require the whole skyflow-common module here to preserve that exact behavior. +const SKYFLOW_ERROR_CODE = require('skyflow-common'); const { default: TokenFormat } = require('../../src/vault/model/options/deidentify-text/token-format'); -const { DetectEntities, Env, TokenMode, OrderByEnum, MaskingMethod } = require('../../src/utils'); +const { DetectEntities, Env, TokenMode, OrderByEnum, MaskingMethod } = require('skyflow-common'); const { Bleep } = require('../../src/vault/model/options/deidentify-file/bleep-audio'); -const { LogLevel } = require('../../src/utils'); +const { LogLevel } = require('skyflow-common'); const { default: Transformations } = require('../../src/vault/model/options/deidentify-text/transformations'); describe('validateDeidentifyFileRequest', () => { @@ -703,7 +709,7 @@ describe('validateSkyflowCredentials', () => { const credentials = { token: 'valid-non-expired-token' }; - jest.spyOn(require('../../src/utils/jwt-utils'), 'isExpired').mockReturnValue(false); + jest.spyOn(require('skyflow-common/lib/utils/jwt-utils'), 'isExpired').mockReturnValue(false); expect(() => validateSkyflowCredentials(credentials)).not.toThrow(); }); @@ -711,7 +717,7 @@ describe('validateSkyflowCredentials', () => { const credentials = { token: 'expired-token' }; - jest.spyOn(require('../../src/utils/jwt-utils'), 'isExpired').mockReturnValue(true); + jest.spyOn(require('skyflow-common/lib/utils/jwt-utils'), 'isExpired').mockReturnValue(true); expect(() => validateSkyflowCredentials(credentials)).toThrow(SKYFLOW_ERROR_CODE.INVALID_BEARER_TOKEN); }); }); @@ -4138,7 +4144,7 @@ describe('validateCredentialsWithId', () => { test('should throw error when token is expired', () => { // Mock isExpired to return true - jest.spyOn(require('../../src/utils/jwt-utils'), 'isExpired').mockReturnValue(true); + jest.spyOn(require('skyflow-common/lib/utils/jwt-utils'), 'isExpired').mockReturnValue(true); const credentials = { token: 'expired-token' @@ -4149,7 +4155,7 @@ describe('validateCredentialsWithId', () => { test('should accept valid token credentials', () => { // Mock isExpired to return false - jest.spyOn(require('../../src/utils/jwt-utils'), 'isExpired').mockReturnValue(false); + jest.spyOn(require('skyflow-common/lib/utils/jwt-utils'), 'isExpired').mockReturnValue(false); const credentials = { token: 'valid-token' @@ -4295,7 +4301,7 @@ describe('validateCredentialsWithId and validateSkyflowCredentials - tokenUri va }); test('validateCredentialsWithId: should accept valid tokenUri (TokenCredentials)', () => { - jest.spyOn(require('../../src/utils/jwt-utils'), 'isExpired').mockReturnValue(false); + jest.spyOn(require('skyflow-common/lib/utils/jwt-utils'), 'isExpired').mockReturnValue(false); const credentials = { token: 'valid-token', tokenUri: validUrl @@ -4359,7 +4365,7 @@ describe('validateCredentialsWithId and validateSkyflowCredentials - tokenUri va }); test('validateSkyflowCredentials: should accept valid tokenUri (TokenCredentials)', () => { - jest.spyOn(require('../../src/utils/jwt-utils'), 'isExpired').mockReturnValue(false); + jest.spyOn(require('skyflow-common/lib/utils/jwt-utils'), 'isExpired').mockReturnValue(false); const credentials = { token: 'valid-token', tokenUri: validUrl diff --git a/test/vault/client/client.test.js b/skyvault/test/vault/client/client.test.js similarity index 99% rename from test/vault/client/client.test.js rename to skyvault/test/vault/client/client.test.js index bfb02efd..24511701 100644 --- a/test/vault/client/client.test.js +++ b/skyvault/test/vault/client/client.test.js @@ -2,9 +2,9 @@ import VaultClient from '../../../src/vault/client'; // Adjust the import path import { Query } from '../../../src/ _generated_/rest/api/resources/query/client/Client'; import { Records } from '../../../src/ _generated_/rest/api/resources/records/client/Client'; import { Tokens } from '../../../src/ _generated_/rest/api/resources/tokens/client/Client'; -import { AuthType, LogLevel, TYPES } from '../../../src/utils'; -import { isExpired } from '../../../src/utils/jwt-utils'; -import SkyflowError from '../../../src/error'; +import { AuthType, LogLevel, TYPES } from 'skyflow-common'; +import { isExpired } from 'skyflow-common'; +import { SkyflowError } from 'skyflow-common'; import { Strings } from '../../../src/ _generated_/rest/api/resources/strings/client/Client'; import { Files } from '../../../src/ _generated_/rest/api/resources/files/client/Client'; @@ -14,8 +14,13 @@ jest.mock('../../../src/ _generated_/rest/api/resources/query/client/Client'); jest.mock('../../../src/ _generated_/rest/api/resources/tokens/client/Client'); jest.mock('../../../src/ _generated_/rest/api/resources/strings/client/Client'); jest.mock('../../../src/ _generated_/rest/api/resources/files/client/Client'); -jest.mock('../../../src/utils/jwt-utils'); -jest.mock('../../../src/error'); +jest.mock('skyflow-common', () => { + const actual = jest.requireActual('skyflow-common'); + return { + ...actual, + isExpired: jest.fn(), + }; +}); describe('VaultClient', () => { const url = 'https://vault.skyflow.com'; diff --git a/test/vault/controller/connection.test.js b/skyvault/test/vault/controller/connection.test.js similarity index 96% rename from test/vault/controller/connection.test.js rename to skyvault/test/vault/controller/connection.test.js index af1441d6..0e545bc9 100644 --- a/test/vault/controller/connection.test.js +++ b/skyvault/test/vault/controller/connection.test.js @@ -10,60 +10,39 @@ import { HTTP_HEADER, CONTENT_TYPE, objectToXML, -} from "../../../src/utils"; +} from "skyflow-common"; import { validateInvokeConnectionRequest } from "../../../src/utils/validations"; import VaultClient from "../../../src/vault/client"; import ConnectionController from "../../../src/vault/controller/connections"; -import SkyflowError from "../../../src/error"; -import SKYFLOW_ERROR_CODE from "../../../src/error/codes"; - -jest.mock("../../../src/utils", () => ({ - fillUrlWithPathAndQueryParams: jest.fn(), - generateSDKMetrics: jest.fn(), - getBearerToken: jest.fn(), - LogLevel: { WARN: 'WARN', INFO: 'INFO', DEBUG: 'DEBUG', ERROR: 'ERROR', OFF: 'OFF' }, - MessageType: { LOG: 'LOG', WARN: 'WARN', ERROR: 'ERROR' }, - RequestMethod: { POST: 'POST', GET: 'GET', PUT: 'PUT', PATCH: 'PATCH' }, - parameterizedString: jest.fn((...args) => args.join(' ')), - printLog: jest.fn(), - SDK: { METRICS_HEADER_KEY: 'sky-metadata' }, - SKYFLOW: { ID: 'skyflowId', LEGACY_ID: 'skyflow_id', AUTH_HEADER_KEY: 'x-skyflow-authorization' }, - REQUEST: { ID_KEY: 'x-request-id' }, - TYPES: { - INSERT: 'INSERT', INSERT_BATCH: 'INSERT_BATCH', DETOKENIZE: 'DETOKENIZE', TOKENIZE: 'TOKENIZE', - DELETE: 'DELETE', UPDATE: 'UPDATE', GET: 'GET', FILE_UPLOAD: 'FILE_UPLOAD', QUERY: 'QUERY', - DETECT: 'DETECT', INVOKE_CONNECTION: 'INVOKE_CONNECTION', DEIDENTIFY_TEXT: 'DEIDENTIFY_TEXT', - REIDENTIFY_TEXT: 'REIDENTIFY_TEXT', DEIDENTIFY_FILE: 'DEIDENTIFY_FILE', DETECT_RUN: 'DETECT_RUN', - }, - HTTP_HEADER: { CONTENT_TYPE: 'Content-Type', CONTENT_TYPE_LOWER: 'content-type', X_REQUEST_ID: 'x-request-id', ERROR_FROM_CLIENT: 'error-from-client' }, - CONTENT_TYPE: { - APPLICATION_JSON: 'application/json', - APPLICATION_X_WWW_FORM_URLENCODED: 'application/x-www-form-urlencoded', - TEXT_PLAIN: 'text/plain', - MULTIPART_FORM_DATA: 'multipart/form-data', - TEXT_XML: 'text/xml', - APPLICATION_XML: 'application/xml', - TEXT_HTML: 'text/html', - }, - objectToXML: jest.fn((obj, root) => `<${root}>${JSON.stringify(obj)}`), -})); -jest.mock("../../../src/utils/logs", () => ({ - __esModule: true, - default: { - infoLogs: { - INVOKE_CONNECTION_TRIGGERED: 'invoke connection triggered', - VALIDATE_CONNECTION_CONFIG: 'validate connection config', - EMIT_REQUEST: 'emit request %s', - INVOKE_CONNECTION_REQUEST_RESOLVED: 'invoke connection request resolved', - }, - errorLogs: { - INVOKE_CONNECTION_REQUEST_REJECTED: 'invoke connection request rejected', - }, - warnLogs: { - DEPRECATED_REQUEST_ID_PROPERTY: 'deprecated request_ID property', +import { SkyflowError } from "skyflow-common"; +import { SKYFLOW_ERROR_CODE } from "skyflow-common"; + +jest.mock("skyflow-common", () => { + const actual = jest.requireActual("skyflow-common"); + return { + ...actual, + fillUrlWithPathAndQueryParams: jest.fn(), + generateSDKMetrics: jest.fn(), + getBearerToken: jest.fn(), + parameterizedString: jest.fn((...args) => args.join(' ')), + printLog: jest.fn(), + objectToXML: jest.fn((obj, root) => `<${root}>${JSON.stringify(obj)}`), + logs: { + infoLogs: { + INVOKE_CONNECTION_TRIGGERED: 'invoke connection triggered', + VALIDATE_CONNECTION_CONFIG: 'validate connection config', + EMIT_REQUEST: 'emit request %s', + INVOKE_CONNECTION_REQUEST_RESOLVED: 'invoke connection request resolved', + }, + errorLogs: { + INVOKE_CONNECTION_REQUEST_REJECTED: 'invoke connection request rejected', + }, + warnLogs: { + DEPRECATED_REQUEST_ID_PROPERTY: 'deprecated request_ID property', + }, }, - }, -})); + }; +}); jest.mock("../../../src/utils/validations"); jest.mock("../../../src/vault/client", () => { return jest.fn().mockImplementation(() => ({})); diff --git a/test/vault/controller/deprecated-detokenize.test.js b/skyvault/test/vault/controller/deprecated-detokenize.test.js similarity index 81% rename from test/vault/controller/deprecated-detokenize.test.js rename to skyvault/test/vault/controller/deprecated-detokenize.test.js index a89ad7fe..1df961d4 100644 --- a/test/vault/controller/deprecated-detokenize.test.js +++ b/skyvault/test/vault/controller/deprecated-detokenize.test.js @@ -1,38 +1,32 @@ import VaultController from '../../../src/vault/controller/vault'; -import { printLog } from '../../../src/utils'; +import { printLog } from 'skyflow-common'; import DetokenizeResponse from '../../../src/vault/model/response/detokenize'; import { validateDetokenizeRequest } from '../../../src/utils/validations'; -jest.mock('../../../src/utils', () => ({ - printLog: jest.fn(), - parameterizedString: jest.fn(), - removeSDKVersion: jest.fn(), - generateSDKMetrics: jest.fn(), - getBearerToken: jest.fn().mockResolvedValue({ key: 'bearer-token' }), - MessageType: { LOG: 'LOG', ERROR: 'ERROR', WARN: 'WARN' }, - LogLevel: { DEBUG: 'DEBUG', INFO: 'INFO', WARN: 'WARN', ERROR: 'ERROR', OFF: 'OFF' }, - TYPES: { DETOKENIZE: 'DETOKENIZE' }, - HTTP_STATUS_CODE: { OK: 200 }, - SDK: { METRICS_HEADER_KEY: 'sky-metadata' }, - SKYFLOW: { ID: 'skyflowId' }, - CONTENT_TYPE: { APPLICATION_JSON: 'application/json' }, - ENCODING_TYPE: { UTF8: 'utf8' }, - RedactionType: { DEFAULT: 'DEFAULT', PLAIN_TEXT: 'PLAIN_TEXT' }, -})); +jest.mock('skyflow-common', () => { + const actual = jest.requireActual('skyflow-common'); + return { + ...actual, + printLog: jest.fn(), + parameterizedString: jest.fn(), + removeSDKVersion: jest.fn(), + generateSDKMetrics: jest.fn(), + getBearerToken: jest.fn().mockResolvedValue({ key: 'bearer-token' }), + logs: { + infoLogs: { CONTROLLER_INITIALIZED: 'init', EMIT_REQUEST: 'emit' }, + errorLogs: { DETOKENIZE_REQUEST_REJECTED: 'rejected' }, + warnLogs: { + DEPRECATED_REQUEST_ID_PROPERTY: "[DEPRECATED] Property 'request_ID' is deprecated and will be removed in an upcoming release. Use 'requestId' instead.", + DEPRECATED_SKYFLOW_ID_PROPERTY: "[DEPRECATED] Property 'skyflow_id' is deprecated.", + }, + }, + }; +}); jest.mock('../../../src/utils/validations', () => ({ validateDetokenizeRequest: jest.fn(), })); -jest.mock('../../../src/utils/logs', () => ({ - infoLogs: { CONTROLLER_INITIALIZED: 'init', EMIT_REQUEST: 'emit' }, - errorLogs: { DETOKENIZE_REQUEST_REJECTED: 'rejected' }, - warnLogs: { - DEPRECATED_REQUEST_ID_PROPERTY: "[DEPRECATED] Property 'request_ID' is deprecated and will be removed in an upcoming release. Use 'requestId' instead.", - DEPRECATED_SKYFLOW_ID_PROPERTY: "[DEPRECATED] Property 'skyflow_id' is deprecated.", - }, -})); - // ─── SHARED SETUP ───────────────────────────────────────────────────────────── const REQ_ID = 'req-detok-001'; diff --git a/test/vault/controller/deprecated-insert.test.js b/skyvault/test/vault/controller/deprecated-insert.test.js similarity index 88% rename from test/vault/controller/deprecated-insert.test.js rename to skyvault/test/vault/controller/deprecated-insert.test.js index 8b256239..0d48dd93 100644 --- a/test/vault/controller/deprecated-insert.test.js +++ b/skyvault/test/vault/controller/deprecated-insert.test.js @@ -1,37 +1,32 @@ import VaultController from '../../../src/vault/controller/vault'; -import { printLog } from '../../../src/utils'; +import { printLog } from 'skyflow-common'; import InsertResponse from '../../../src/vault/model/response/insert'; import { validateInsertRequest } from '../../../src/utils/validations'; -jest.mock('../../../src/utils', () => ({ - printLog: jest.fn(), - parameterizedString: jest.fn(), - removeSDKVersion: jest.fn(), - generateSDKMetrics: jest.fn(), - getBearerToken: jest.fn().mockResolvedValue({ key: 'bearer-token' }), - MessageType: { LOG: 'LOG', ERROR: 'ERROR', WARN: 'WARN' }, - LogLevel: { DEBUG: 'DEBUG', INFO: 'INFO', WARN: 'WARN', ERROR: 'ERROR', OFF: 'OFF' }, - TYPES: { INSERT: 'INSERT', INSERT_BATCH: 'INSERT_BATCH' }, - HTTP_STATUS_CODE: { OK: 200, BAD_REQUEST: 400 }, - SDK: { METRICS_HEADER_KEY: 'sky-metadata' }, - SKYFLOW: { ID: 'skyflowId' }, - CONTENT_TYPE: { APPLICATION_JSON: 'application/json' }, - ENCODING_TYPE: { UTF8: 'utf8' }, -})); +jest.mock('skyflow-common', () => { + const actual = jest.requireActual('skyflow-common'); + return { + ...actual, + printLog: jest.fn(), + parameterizedString: jest.fn(), + removeSDKVersion: jest.fn(), + generateSDKMetrics: jest.fn(), + getBearerToken: jest.fn().mockResolvedValue({ key: 'bearer-token' }), + logs: { + infoLogs: { CONTROLLER_INITIALIZED: 'init', EMIT_REQUEST: 'emit' }, + errorLogs: { INSERT_REQUEST_REJECTED: 'rejected' }, + warnLogs: { + DEPRECATED_SKYFLOW_ID_PROPERTY: "[DEPRECATED] Property 'skyflow_id' is deprecated and will be removed in an upcoming release. Use 'skyflowId' instead.", + DEPRECATED_REQUEST_ID_PROPERTY: "[DEPRECATED] Property 'request_ID' is deprecated and will be removed in an upcoming release. Use 'requestId' instead.", + }, + }, + }; +}); jest.mock('../../../src/utils/validations', () => ({ validateInsertRequest: jest.fn(), })); -jest.mock('../../../src/utils/logs', () => ({ - infoLogs: { CONTROLLER_INITIALIZED: 'init', EMIT_REQUEST: 'emit' }, - errorLogs: { INSERT_REQUEST_REJECTED: 'rejected' }, - warnLogs: { - DEPRECATED_SKYFLOW_ID_PROPERTY: "[DEPRECATED] Property 'skyflow_id' is deprecated and will be removed in an upcoming release. Use 'skyflowId' instead.", - DEPRECATED_REQUEST_ID_PROPERTY: "[DEPRECATED] Property 'request_ID' is deprecated and will be removed in an upcoming release. Use 'requestId' instead.", - }, -})); - // ─── SHARED SETUP ───────────────────────────────────────────────────────────── function makeClient(overrides = {}) { diff --git a/test/vault/controller/detect.test.js b/skyvault/test/vault/controller/detect.test.js similarity index 97% rename from test/vault/controller/detect.test.js rename to skyvault/test/vault/controller/detect.test.js index c01dccb1..5d55678d 100644 --- a/test/vault/controller/detect.test.js +++ b/skyvault/test/vault/controller/detect.test.js @@ -7,7 +7,7 @@ import DeidentifyFileOptions from '../../../src/vault/model/options/deidentify-f import TokenFormat from '../../../src/vault/model/options/deidentify-text/token-format'; import Transformations from '../../../src/vault/model/options/deidentify-text/transformations'; import { Bleep } from '../../../src/vault/model/options/deidentify-file/bleep-audio'; -import { DETECT_STATUS, ENCODING_TYPE, TYPES } from '../../../src/utils'; +import { DETECT_STATUS, ENCODING_TYPE, TYPES } from 'skyflow-common'; import fs from 'fs'; function createFullDeidentifyFileOptions() { @@ -42,76 +42,17 @@ function createFullDeidentifyFileOptions() { return options; } -jest.mock('../../../src/utils', () => ({ - printLog: jest.fn(), - parameterizedString: jest.fn(), - removeSDKVersion: jest.fn(), - MessageType: { - LOG: 'LOG', - ERROR: 'ERROR', - WARN: 'WARN', - }, - LogLevel: { - DEBUG: 'DEBUG', - INFO: 'INFO', - WARN: 'WARN', - ERROR: 'ERROR', - OFF: 'OFF', - }, - TYPES: { - DETECT: 'DETECT', - DEIDENTIFY_FILE: 'DEIDENTIFY_FILE', - DETECT_RUN: 'DETECT_RUN', - DEIDENTIFY_TEXT: 'DEIDENTIFY_TEXT', - REIDENTIFY_TEXT: 'REIDENTIFY_TEXT', - }, - DeidenitfyFileRequestTypes: { - AUDIO: 'AUDIO', - TEXT: 'TEXT', - PDF: 'PDF', - IMAGE: 'IMAGE', - PPT: 'PPT', - SPREADSHEET: 'SPREADSHEET', - STRUCTURED_TEXT: 'STRUCTURED_TEXT', - DOCUMENT: 'DOCUMENT', - FILE: 'FILE', - }, - DETECT_STATUS: { - IN_PROGRESS: 'IN_PROGRESS', - SUCCESS: 'SUCCESS', - FAILED: 'FAILED', - }, - SDK: { - METRICS_HEADER_KEY: 'sky-metadata', - }, - FILE_EXTENSION: { - JSON: 'json', - MP3: 'mp3', - WAV: 'wav', - }, - FILE_FORMAT_TYPE: { - TXT: 'txt', - PDF: 'pdf', - }, - FILE_PROCESSING: { - PROCESSED_PREFIX: 'processed-', - DEIDENTIFIED_PREFIX: 'deidentified.', - ENTITIES: 'entities', - }, - ENCODING_TYPE: { - UTF8: 'utf8', - BASE64: 'base64', - BINARY: 'binary', - UTF_8: 'utf-8', - }, - TokenType: { - ENTITY_UNIQUE_COUNTER: 'entity_unq_counter', - ENTITY_ONLY: 'entity_only', - VAULT_TOKEN: 'vault_token', - }, - generateSDKMetrics: jest.fn().mockReturnValue({ sdk: 'metrics' }), - getBearerToken: jest.fn().mockResolvedValue(Promise.resolve('your-bearer-token')), -})); +jest.mock('skyflow-common', () => { + const actual = jest.requireActual('skyflow-common'); + return { + ...actual, + printLog: jest.fn(), + parameterizedString: jest.fn(), + removeSDKVersion: jest.fn(), + generateSDKMetrics: jest.fn().mockReturnValue({ sdk: 'metrics' }), + getBearerToken: jest.fn().mockResolvedValue(Promise.resolve('your-bearer-token')), + }; +}); jest.mock('../../../src/utils/validations', () => ({ validateDeIdentifyTextRequest: jest.fn(), diff --git a/test/vault/controller/vault.test.js b/skyvault/test/vault/controller/vault.test.js similarity index 98% rename from test/vault/controller/vault.test.js rename to skyvault/test/vault/controller/vault.test.js index 3d2cc56e..68565fa5 100644 --- a/test/vault/controller/vault.test.js +++ b/skyvault/test/vault/controller/vault.test.js @@ -1,6 +1,6 @@ import VaultController from '../../../src/vault/controller/vault'; -import { printLog, MessageType, removeSDKVersion, LogLevel } from '../../../src/utils'; -import logs from '../../../src/utils/logs'; +import { printLog, MessageType, removeSDKVersion, LogLevel } from 'skyflow-common'; +import { logs } from 'skyflow-common'; import { validateInsertRequest, validateDetokenizeRequest, validateDeleteRequest, validateTokenizeRequest, validateQueryRequest, validateUpdateRequest, validateUploadFileRequest, validateGetRequest, validateGetColumnRequest } from '../../../src/utils/validations'; import InsertResponse from '../../../src/vault/model/response/insert'; import DeleteResponse from '../../../src/vault/model/response/delete'; @@ -11,7 +11,7 @@ import FileUploadResponse from '../../../src/vault/model/response/file-upload'; import GetResponse from '../../../src/vault/model/response/get'; import GetRequest from '../../../src/vault/model/request/get'; import GetColumnRequest from '../../../src/vault/model/request/get-column'; -import SkyflowError from '../../../src/error'; +import { SkyflowError } from 'skyflow-common'; import * as fs from 'fs'; jest.mock('fs', () => ({ @@ -41,64 +41,29 @@ if (typeof File === 'undefined') { }; } -jest.mock('../../../src/utils', () => ({ - printLog: jest.fn(), - parameterizedString: jest.fn(), - removeSDKVersion: jest.fn(), - MessageType: { - LOG: 'LOG', - ERROR: 'ERROR', - WARN: 'WARN', - }, - RedactionType: { - DEFAULT: 'DEFAULT', - PLAIN_TEXT: 'PLAIN_TEXT', - MASKED: 'MASKED', - REDACTED: 'REDACTED', - }, - SDK: { - METRICS_HEADER_KEY: 'sky-metadata', - }, - SKYFLOW: { - ID: 'skyflowId', - LEGACY_ID: 'skyflow_id', - }, - CONTENT_TYPE: { - APPLICATION_JSON: 'application/json', - APPLICATION_X_WWW_FORM_URLENCODED: 'application/x-www-form-urlencoded', - TEXT_PLAIN: 'text/plain', - }, - ENCODING_TYPE: { - UTF8: 'utf8', - BASE64: 'base64', - BINARY: 'binary', - UTF_8: 'utf-8', - }, - TYPES: { - INSERT: 'INSERT', - INSERT_BATCH: 'INSERT_BATCH', - DETOKENIZE: 'DETOKENIZE', - TOKENIZE: 'TOKENIZE', - DELETE: 'DELETE', - UPDATE: 'UPDATE', - GET: 'GET', - FILE_UPLOAD: 'FILE_UPLOAD', - QUERY: 'QUERY', - INVOKE_CONNECTION: 'INVOKE_CONNECTION', - }, - generateSDKMetrics: jest.fn(), - getBearerToken: jest.fn().mockResolvedValue(Promise.resolve('your-bearer-token')), - HTTP_STATUS_CODE: { - OK: 200, - BAD_REQUEST: 400, - INTERNAL_SERVER_ERROR: 500, - }, - HTTP_HEADER: { - CONTENT_TYPE: 'Content-Type', - X_REQUEST_ID: 'x-request-id', - }, - SkyflowRecordError: {}, -})); +jest.mock('skyflow-common', () => { + const actual = jest.requireActual('skyflow-common'); + return { + ...actual, + printLog: jest.fn(), + parameterizedString: jest.fn(), + removeSDKVersion: jest.fn(), + generateSDKMetrics: jest.fn(), + getBearerToken: jest.fn().mockResolvedValue(Promise.resolve('your-bearer-token')), + logs: { + infoLogs: { + CONTROLLER_INITIALIZED: 'VaultController initialized successfully.', + }, + errorLogs: { + INSERT_REQUEST_REJECTED: 'INSERT_REJECTED', + }, + warnLogs: { + DEPRECATED_SKYFLOW_ID_PROPERTY: "[DEPRECATED] Property 'skyflow_id' is deprecated and will be removed in an upcoming release. Use 'skyflowId' instead.", + DEPRECATED_REQUEST_ID_PROPERTY: "[DEPRECATED] Property 'request_ID' is deprecated and will be removed in an upcoming release. Use 'requestId' instead.", + }, + }, + }; +}); jest.mock('../../../src/utils/validations', () => ({ validateInsertRequest: jest.fn(), @@ -112,19 +77,6 @@ jest.mock('../../../src/utils/validations', () => ({ validateGetColumnRequest: jest.fn(), })); -jest.mock('../../../src/utils/logs', () => ({ - infoLogs: { - CONTROLLER_INITIALIZED: 'VaultController initialized successfully.', - }, - errorLogs: { - INSERT_REQUEST_REJECTED: 'INSERT_REJECTED', - }, - warnLogs: { - DEPRECATED_SKYFLOW_ID_PROPERTY: "[DEPRECATED] Property 'skyflow_id' is deprecated and will be removed in an upcoming release. Use 'skyflowId' instead.", - DEPRECATED_REQUEST_ID_PROPERTY: "[DEPRECATED] Property 'request_ID' is deprecated and will be removed in an upcoming release. Use 'requestId' instead.", - }, -})); - describe('VaultController', () => { let mockVaultClient; diff --git a/test/vault/model/deprecated.test.js b/skyvault/test/vault/model/deprecated.test.js similarity index 96% rename from test/vault/model/deprecated.test.js rename to skyvault/test/vault/model/deprecated.test.js index 8fbf655d..1169c62b 100644 --- a/test/vault/model/deprecated.test.js +++ b/skyvault/test/vault/model/deprecated.test.js @@ -1,23 +1,12 @@ -jest.mock("../../../src/utils", () => ({ - printLog: jest.fn(), - MessageType: { LOG: "LOG", ERROR: "ERROR", WARN: "WARN" }, - LogLevel: { - DEBUG: "DEBUG", - INFO: "INFO", - WARN: "WARN", - ERROR: "ERROR", - OFF: "OFF", - }, - OrderByEnum: { ASC: "ASC", DESC: "DESC" }, - RedactionType: { - DEFAULT: "DEFAULT", - PLAIN_TEXT: "PLAIN_TEXT", - MASKED: "MASKED", - REDACTED: "REDACTED", - }, -})); - -import { printLog, OrderByEnum, RedactionType } from "../../../src/utils"; +jest.mock("skyflow-common", () => { + const actual = jest.requireActual("skyflow-common"); + return { + ...actual, + printLog: jest.fn(), + }; +}); + +import { printLog, OrderByEnum, RedactionType } from "skyflow-common"; import DetokenizeOptions from "../../../src/vault/model/options/detokenize"; import GetOptions from "../../../src/vault/model/options/get"; import FileUploadRequest from "../../../src/vault/model/request/file-upload"; diff --git a/test/vault/skyflow/skyflow.test.js b/skyvault/test/vault/skyflow/skyflow.test.js similarity index 100% rename from test/vault/skyflow/skyflow.test.js rename to skyvault/test/vault/skyflow/skyflow.test.js diff --git a/skyvault/tsconfig.dts.json b/skyvault/tsconfig.dts.json new file mode 100644 index 00000000..8d46eaf5 --- /dev/null +++ b/skyvault/tsconfig.dts.json @@ -0,0 +1,25 @@ +{ + "compilerOptions": { + "target": "es2019", + "module": "commonjs", + "moduleResolution": "node", + "lib": ["es2015", "es2017", "es2019", "dom"], + "esModuleInterop": true, + "allowSyntheticDefaultImports": true, + "allowJs": true, + "strict": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": false, + "noImplicitAny": false, + "declaration": true, + "emitDeclarationOnly": true, + "composite": false, + "incremental": false, + "stripInternal": true, + "types": ["node"], + "rootDir": "src" + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "lib", "dist", "test"] +} diff --git a/skyvault/tsconfig.json b/skyvault/tsconfig.json new file mode 100644 index 00000000..c228d95f --- /dev/null +++ b/skyvault/tsconfig.json @@ -0,0 +1,30 @@ +{ + "compilerOptions": { + "composite": true, + "sourceMap": true, + "declaration": true, + "declarationMap": true, + "stripInternal": true, + "noImplicitAny": false, + "target": "es6", + "lib": ["es2015", "es2017", "es2019", "dom"], + "esModuleInterop": true, + "module": "commonjs", + "moduleResolution": "node", + "removeComments": false, + "allowSyntheticDefaultImports": true, + "allowJs": true, + "strict": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": false, + "isolatedModules": false, + "noEmit": false, + "rootDir": "src", + "outDir": "lib", + "types": ["node"] + }, + "references": [{ "path": "../common" }], + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "lib", "test"] +} diff --git a/skyvault/tsup.config.ts b/skyvault/tsup.config.ts new file mode 100644 index 00000000..067ef014 --- /dev/null +++ b/skyvault/tsup.config.ts @@ -0,0 +1,19 @@ +import { defineConfig } from 'tsup'; + +// Bundles the published entry, inlining the private `skyflow-common` workspace +// package so the tarball is self-contained (no skyflow-common dependency ships). +// Real npm dependencies stay external (declared in package.json "dependencies"). +export default defineConfig({ + entry: { index: 'src/index.ts' }, + format: ['cjs'], + outDir: 'dist', + target: 'es2019', + platform: 'node', + noExternal: ['skyflow-common'], + tsconfig: 'tsconfig.dts.json', + dts: true, + sourcemap: true, + clean: true, + splitting: false, + shims: false, +}); diff --git a/src/error/messages/index.ts b/src/error/messages/index.ts deleted file mode 100644 index 2b634c84..00000000 --- a/src/error/messages/index.ts +++ /dev/null @@ -1,252 +0,0 @@ -import sdkDetails from "../../../package.json"; - -const errorPrefix = `Skyflow Node SDK v${sdkDetails.version}`; - -const errorMessages = { - CONFIG_MISSING: `${errorPrefix} Initialization failed. Skyflow config cannot be empty. Specify a valid skyflow config.`, - INVALID_SKYFLOW_CONFIG: `${errorPrefix} Initialization failed. Invalid skyflow config. Vault/Connection config key missing in skyflow config.`, - INVALID_TYPE_FOR_CONFIG: `${errorPrefix} Initialization failed. Invalid %s1 config. Specify a valid %s1 config.`, - EMPTY_VAULT_CONFIG: `${errorPrefix} Initialization failed. Vault config cannot be empty. Specify a valid vault config.`, - EMPTY_CONNECTION_CONFIG: `${errorPrefix} Initialization failed. Connection config cannot be empty. Specify a valid connection config.`, - - EMPTY_VAULT_ID: `${errorPrefix} Initialization failed. Invalid vault ID. Specify a valid vault Id.`, - EMPTY_VAULT_ID_VALIDATION: `${errorPrefix} Validation error. Invalid vault ID. Specify a valid vault Id.`, - INVALID_VAULT_ID: `${errorPrefix} Initialization failed. Invalid vault ID. Specify a valid vault Id as a string.`, - EMPTY_CLUSTER_ID: `${errorPrefix} Initialization failed. Invalid cluster ID. Specify a valid cluster Id for vault with vaultId %s1 .`, - INVALID_CLUSTER_ID: `${errorPrefix} Initialization failed. Invalid cluster ID. Specify cluster Id as a string for vault with vaultId %s1.`, - INVALID_TOKEN: `${errorPrefix} Validation error. Invalid token. Specify a valid token.`, - TOKEN_EXPIRED: `${errorPrefix} Validation error. Token provided is either invalid or has expired. Specify a valid token.`, - INVALID_ENV: `${errorPrefix} Initialization failed. Invalid env. Specify a valid env for vault with vaultId %s1.`, - INVALID_LOG_LEVEL: `${errorPrefix} Initialization failed. Invalid log level. Specify a valid log level.`, - EMPTY_CREDENTIAL_FILE_PATH: `${errorPrefix} Initialization failed. Invalid credentials. Specify a valid file path.`, - INVALID_CREDENTIAL_FILE_PATH: `${errorPrefix} Initialization failed. Invalid credentials. Expected file path to be a string.`, - - INVALID_CREDENTIALS_FILE_PATH: `${errorPrefix} Initialization failed. Invalid skyflow credentials. Expected file path to exists.`, - INVALID_TOKEN_URI: `${errorPrefix} Initialization failed. Invalid Skyflow credentials. The token URI must be a string and a valid URL.`, - INVALID_TOKEN_URI_WITH_ID: `${errorPrefix} Initialization failed. Invalid Skyflow credentials. The token URI must be a string and a valid URL for %s1 with %s2 %s3.`, - INVALID_KEY: `${errorPrefix} Initialization failed. Invalid skyflow credentials. Specify a valid api key.`, - INVALID_PARSED_CREDENTIALS_STRING: `${errorPrefix} Initialization failed. Invalid skyflow credentials. Specify a valid credentials string.`, - INVALID_BEARER_TOKEN: `${errorPrefix} Initialization failed. Invalid skyflow credentials. Bearer token is invalid or expired. Specify a valid token.`, - - INVALID_FILE_PATH_WITH_ID: `${errorPrefix} Initialization failed. Invalid credentials. Expected file path to exists for %s1 with %s2 %s3.`, - INVALID_KEY_WITH_ID: `${errorPrefix} Initialization failed. Invalid credentials. Specify a valid api key for %s1 with %s2 %s3.`, - INVALID_PARSED_CREDENTIALS_STRING_WITH_ID: `${errorPrefix} Initialization failed. Invalid credentials. Specify a valid credentials string for %s1 with %s2 %s3.`, - INVALID_BEARER_TOKEN_WITH_ID: `${errorPrefix} Initialization failed. Invalid credentials. Bearer token is invalid or expired. Specify a valid token for %s1 with %s2 %s3.`, - - EMPTY_CONNECTION_ID_VALIDATION: `${errorPrefix} Validation error. Invalid connection ID. Specify a valid connection Id.`, - EMPTY_CONNECTION_ID: `${errorPrefix} Initialization failed. Invalid connection ID. Specify a valid connection Id.`, - INVALID_CONNECTION_ID: `${errorPrefix} Initialization failed. Invalid connection ID. Specify connection Id as a string.`, - EMPTY_CONNECTION_URL: `${errorPrefix} Initialization failed. Invalid connection URL. Specify a valid connection Url.`, - INVALID_CONNECTION_URL: `${errorPrefix} Initialization failed. Invalid connection URL. Specify connection Url as a valid url.`, - - VAULT_ID_EXITS_IN_CONFIG_LIST: `${errorPrefix} Validation error. %s1 already exists in the config list. Specify a new vaultId.`, - CONNECTION_ID_EXITS_IN_CONFIG_LIST: `${errorPrefix} Validation error. %s1 already exists in the config list. Specify a new vaultId.`, - VAULT_ID_NOT_IN_CONFIG_LIST: `${errorPrefix} Validation error. %s1 is missing from the config. Specify the vaultId's from config.`, - CONNECTION_ID_NOT_IN_CONFIG_LIST: `${errorPrefix} Validation error. %s1 is missing from the config. Specify the connectionIds from config.`, - - EMPTY_CREDENTIALS: `${errorPrefix} Validation error. Invalid credentials. Credentials must not be empty.`, - INVALID_CREDENTIALS: `${errorPrefix} Validation error. Invalid credentials. Specify a valid credentials.`, - CREDENTIALS_WITH_NO_VALID_KEY: `${errorPrefix} Validation error. Invalid credentials. Credentials must include one of the following: { apiKey, token, credentials, path }.`, - MULTIPLE_CREDENTIALS_PASSED: `${errorPrefix} Validation error. Multiple credentials provided. Specify only one of the following: { apiKey, token, credentials, path }.`, - INVALID_CREDENTIALS_WITH_ID: `${errorPrefix} Validation error. Invalid credentials. Credentials must include one of the following: { apiKey, token, credentials, path } for %s1 with %s2 %s3.`, - MULTIPLE_CREDENTIALS_PASSED_WITH_ID: `${errorPrefix} Validation error. Invalid credentials.Specify only one of the following: { apiKey, token, credentials, path } for %s1 with %s2 %s3.`, - - FILE_NOT_FOUND: `${errorPrefix} Validation error. Credential file not found at %s1. Verify the file path.`, - INVALID_JSON_FILE: `${errorPrefix} Validation error. File at %s1 is not in valid JSON format. Verify the file contents.`, - - EMPTY_CREDENTIALS_STRING: `${errorPrefix} Validation error. Invalid credentials. Specify a valid credentials.`, - INVALID_CREDENTIALS_STRING: `${errorPrefix} Validation error. Invalid credentials. Specify credentials as a string.`, - - MISSING_PRIVATE_KEY: `${errorPrefix} Validation error. Unable to read private key in credentials. Verify your private key.`, - MISSING_CLIENT_ID: `${errorPrefix} Validation error. Unable to read client ID in credentials. Verify your client ID.`, - MISSING_KEY_ID: `${errorPrefix} Validation error. Unable to read key ID in credentials. Verify your key ID.`, - MISSING_TOKEN_URI: `${errorPrefix} Validation error. Unable to read token URI in credentials. Verify your token URI.`, - - INVALID_ROLES_KEY_TYPE: `${errorPrefix} Validation error. Invalid roles. Specify roles as an array.`, - INVALID_CONTEXT: `${errorPrefix} Validation error. Invalid context. Specify context as a string.`, - EMPTY_ROLES: `${errorPrefix} Validation error. Invalid roles. Specify at least one role.`, - - INVALID_JSON_FORMAT: `${errorPrefix} Validation error. Credentials is not in valid JSON format. Verify the credentials.`, - - EMPTY_DATA_TOKENS: `${errorPrefix} Validation error. Invalid data tokens. Specify valid data tokens.`, - DATA_TOKEN_KEY_TYPE: `${errorPrefix} Validation error. Invalid data tokens. Specify data token as an string array.`, - TIME_TO_LIVE_KET_TYPE: `${errorPrefix} Validation error. Invalid time to live. Specify time to live parameter as an string.`, - - INVALID_DELETE_IDS_INPUT: `${errorPrefix} Validation error. Invalid delete ids type in delete request. Specify delete ids as a string array.`, - EMPTY_DELETE_IDS: `${errorPrefix} Validation error. Delete ids array cannot be empty. Specify id's in delete request.`, - INVALID_ID_IN_DELETE: `${errorPrefix} Validation error. Invalid type of id passed in delete request. Id must be of type string at index %s1.`, - INVALID_DELETE_REQUEST: `${errorPrefix} Validation error. Invalid delete request. Specify a valid delete request.`, - EMPTY_ID_IN_DELETE: `${errorPrefix} Validation error. Id cannot be empty in delete request. Specify a valid id.`, - - MISSING_REDACTION_TYPE_IN_DETOKENIZE: `${errorPrefix} Validation error. Redaction type cannot be empty in detokenize request. Specify the redaction type.`, - INVALID_REDACTION_TYPE_IN_DETOKENIZE: `${errorPrefix} Validation error. Invalid redaction type in detokenize request. Specify a redaction type.`, - INVALID_TOKENS_TYPE_IN_DETOKENIZE: `${errorPrefix} Validation error. Invalid tokens type in detokenize request. Specify tokens as a string array.`, - EMPTY_TOKENS_IN_DETOKENIZE: `${errorPrefix} Validation error. Tokens array cannot be empty. Specify token's in detokenize request.`, - EMPTY_TOKEN_IN_DETOKENIZE: `${errorPrefix} Validation error. Token cannot be empty in detokenize request. Specify a valid token at index %s1.`, - INVALID_TOKEN_IN_DETOKENIZE: `${errorPrefix} Validation error. Invalid type of token passed in detokenize request. token must be of type string at index %s1.`, - INVALID_DETOKENIZE_REQUEST: `${errorPrefix} Validation error. Invalid detokenize request. Specify a valid detokenize request.`, - - MISSING_VALUES_IN_TOKENIZE: `${errorPrefix} Validation error. Values cannot be empty in tokenize request. Specify valid values.`, - INVALID_VALUES_TYPE_IN_TOKENIZE: `${errorPrefix} Validation error. Invalid values type in tokenize request. Specify valid values of type array.`, - EMPTY_VALUES_IN_TOKENIZE: `${errorPrefix} Validation error. Values array cannot be empty. Specify value's in tokenize request.`, - EMPTY_DATA_IN_TOKENIZE: `${errorPrefix} Validation error. Data cannot be empty in tokenize request. Specify a valid data at index %s1.`, - INVALID_DATA_IN_TOKENIZE: `${errorPrefix} Validation error. Invalid Data. Specify a valid data at index %s1.`, - INVALID_COLUMN_GROUP_IN_TOKENIZE: `${errorPrefix} Validation error. Invalid type of column group passed in tokenize request. Column group must be of type string at index %s1.`, - INVALID_VALUE_IN_TOKENIZE: `${errorPrefix} Validation error. Invalid type of value passed in tokenize request. Value must be of type string at index %s1.`, - INVALID_TOKENIZE_REQUEST: `${errorPrefix} Validation error. Invalid tokenize request. Specify a valid tokenize request.`, - EMPTY_COLUMN_GROUP_IN_TOKENIZE: `${errorPrefix} Validation error. Column group cannot be empty in tokenize request. Specify a valid column group at index %s1.`, - EMPTY_VALUE_IN_TOKENIZE: `${errorPrefix} Validation error. Value cannot be empty in tokenize request. Specify a valid value at index %s1.`, - - INVALID_RECORD_IN_INSERT: `${errorPrefix} Validation error. Invalid data in insert request. data must be of type object at index %s1.`, - INVALID_RECORD_IN_UPDATE: `${errorPrefix} Validation error. Invalid data in update request. data must be of type object.`, - EMPTY_RECORD_IN_INSERT: `${errorPrefix} Validation error. Data cannot be empty in insert request. Specify valid data at index %s1.`, - INVALID_INSERT_REQUEST: `${errorPrefix} Validation error. Invalid insert request. Specify a valid insert request.`, - INVALID_TYPE_OF_DATA_IN_INSERT: `${errorPrefix} Validation error. Invalid type of data in insert request. Specify data as a object array.`, - EMPTY_DATA_IN_INSERT: `${errorPrefix} Validation error. Data array cannot be empty. Specify data in insert request.`, - - INVALID_QUERY_REQUEST: `${errorPrefix} Validation error. Invalid query request. Specify a valid query request.`, - EMPTY_QUERY: `${errorPrefix} Validation error. Query cannot be empty in query request. Specify a valid query.`, - INVALID_QUERY: `${errorPrefix} Validation error. Invalid query in query request. Specify a valid query.`, - - INVALID_FILE_UPLOAD_REQUEST: `${errorPrefix} Validation error. Invalid file upload request. Specify a valid file upload request.`, - MISSING_TABLE_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Table name cannot be empty in file upload request. Specify table name as a string.`, - INVALID_TABLE_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Invalid table name in file upload request. Specify a valid table name.`, - MISSING_SKYFLOW_ID_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Skyflow id cannot be empty in file upload request. Specify a valid skyflow Id as string.`, - INVALID_SKYFLOW_ID_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Invalid skyflow Id in file upload request. Specify a valid skyflow Id.`, - MISSING_COLUMN_NAME_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Column name cannot be empty in file upload request. Specify a valid column name as string.`, - INVALID_COLUMN_NAME_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Invalid column name in file upload request. Specify a valid column name.`, - MISSING_FILE_PATH_IN_UPLOAD_FILE: `${errorPrefix} Validation error. File path cannot be empty in file upload request. Specify a valid file path as string.`, - INVALID_FILE_PATH_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Invalid file path in file upload request. Specify a valid file path.`, - MISSING_FILE_SOURCE_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Provide exactly one of filePath, base64, or fileObject.`, - MISSING_FILE_NAME_FOR_BASE64: `${errorPrefix} Validation error. File name is required when providing a base64 string`, - INVALID_FILE_OBJECT_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Invalid file object in file upload request. Specify a valid file object.`, - MISSING_FILE_NAME_IN_FILE_OBJECT: `${errorPrefix} Validation error. File name is required when providing a file object. Specify a valid file name.`, - INVALID_BASE64_IN_UPLOAD_FILE: `${errorPrefix} Validation error. Invalid base64 string in file upload request. Specify a valid base64 string.`, - - MISSING_SKYFLOW_ID_IN_UPDATE: `${errorPrefix} Validation error. Skyflow id name cannot be empty in update request. Specify a skyflow Id name as string.`, - INVALID_SKYFLOW_ID_IN_UPDATE: `${errorPrefix} Validation error. Invalid skyflow Id in update request. Specify a valid skyflow Id.`, - INVALID_TYPE_OF_UPDATE_DATA: `${errorPrefix} Validation error. Invalid update data in update request. Specify a valid update data as array of object.`, - EMPTY_UPDATE_DATA: `${errorPrefix} Validation error. Update data cannot be empty in update request. Specify a valid update data.`, - INVALID_UPDATE_REQUEST: `${errorPrefix} Validation error. Invalid update request. Specify a valid update request.`, - INVALID_DATA_IN_UPDATE: `${errorPrefix} Validation error. Invalid data in update request. data must be of type object at index %s1.`, - EMPTY_DATA_IN_UPDATE: `${errorPrefix} Validation error. Data cannot be empty in update request. Specify a valid data at index %s1.`, - INVALID_UPDATE_TOKENS: `${errorPrefix} Validation error. Invalid tokens. Specify valid tokens as object.`, - INVALID_TOKEN_IN_UPDATE: `${errorPrefix} Validation error. Invalid tokens. Specify valid tokens as key value pairs.`, - - EMPTY_TABLE_NAME: `${errorPrefix} Validation error. Table name cannot be empty. Specify a valid table name.`, - INVALID_TABLE_NAME: `${errorPrefix} Validation error. Invalid table name. Specify a valid table name as string.`, - EMPTY_REDACTION_TYPE: `${errorPrefix} Validation error. Redaction type cannot be empty. Specify a valid redaction type.`, - INVALID_REDACTION_TYPE: `${errorPrefix} Validation error. Invalid redaction type. Specify a valid redaction type.`, - - INVALID_TYPE_OF_IDS: `${errorPrefix} Validation error. Invalid ids passed in get request. Specify valid ids as array of string.`, - EMPTY_IDS_IN_GET: `${errorPrefix} Validation error. Ids cannot be empty in get request. Specify valid ids.`, - EMPTY_ID_IN_GET: `${errorPrefix} Validation error. Id cannot be empty. Specify a valid Id at index %s1.`, - INVALID_ID_IN_GET: `${errorPrefix} Validation error. Invalid Id. Specify a valid Id at index %s1 as string.`, - INVALID_GET_REQUEST: `${errorPrefix} Validation error. Invalid get request. Specify a valid get request.`, - - EMPTY_COLUMN_NAME: `${errorPrefix} Validation error. Column name cannot be empty. Specify a valid column name.`, - INVALID_COLUMN_NAME: `${errorPrefix} Validation error. Invalid column name. Specify a valid column name as string.`, - INVALID_COLUMN_VALUES: `${errorPrefix} Validation error. Invalid column values. Specify valid column values as string array.`, - EMPTY_COLUMN_VALUES: `${errorPrefix} Validation error. Column values cannot be empty. Specify valid column values.`, - EMPTY_COLUMN_VALUE: `${errorPrefix} Validation error. Column value cannot be empty. Specify a valid column value at index %s1.`, - INVALID_COLUMN_VALUE: `${errorPrefix} Validation error. Invalid column value. Specify a valid column value at index %s1 as string.`, - INVALID_GET_COLUMN_REQUEST: `${errorPrefix} Validation error. Invalid get column request. Specify a valid get column request.`, - - EMPTY_URL: `${errorPrefix} Validation error. Url cannot be empty. Specify a valid url.`, - INVALID_URL: `${errorPrefix} Validation error. Invalid url. Specify a valid url as a string.`, - EMPTY_METHOD_NAME: `${errorPrefix} Validation error. Method name cannot be empty. Specify a valid method name.`, - INVALID_METHOD_NAME: `${errorPrefix} Validation error. Invalid method name. Specify a valid method name as a string.`, - - EMPTY_PATH_PARAMS: `${errorPrefix} Validation error. Path params cannot be empty. Specify valid path params.`, - INVALID_PATH_PARAMS: `${errorPrefix} Validation error. Invalid path params. Specify valid path params.`, - EMPTY_QUERY_PARAMS: `${errorPrefix} Validation error. Query params cannot be empty. Specify valid query params.`, - INVALID_QUERY_PARAMS: `${errorPrefix} Validation error. Invalid query params. Specify valid query params.`, - EMPTY_BODY: `${errorPrefix} Validation error. Body cannot be empty. Specify a valid body.`, - INVALID_BODY: `${errorPrefix} Validation error. Invalid body. Specify a valid body.`, - EMPTY_HEADERS: `${errorPrefix} Validation error. Headers cannot be empty. Specify valid headers.`, - INVALID_HEADERS: `${errorPrefix} Validation error. Invalid headers. Specify valid headers.`, - - INVALID_INVOKE_CONNECTION_REQUEST: `${errorPrefix} Validation error. Invalid invoke connection request. Specify a valid get invoke connection request.`, - ERROR_OCCURRED: `${errorPrefix} API error. Error occurred.`, - - EMPTY_INSERT_TOKEN: `${errorPrefix} Validation error. Tokens object cannot be empty. Specify a valid tokens object at index %s1.`, - INVALID_INSERT_TOKEN: `${errorPrefix} Validation error. Invalid tokens object. Specify a valid tokens object at index %s1.`, - INVALID_INSERT_TOKENS: `${errorPrefix} Validation error. Invalid tokens. Specify valid tokens as object array.`, - INVALID_TOKEN_MODE: `${errorPrefix} Validation error. The token mode key has a value of type %s1. Specify token as type of TokenMode.`, - INVALID_HOMOGENEOUS: `${errorPrefix} Validation error. The homogeneous key has a value of type %s1. Specify homogeneous as boolean.`, - INVALID_TOKEN_STRICT: `${errorPrefix} Validation error. The tokenStrict key has a value of type %s1. Specify tokenStrict as boolean.`, - INVALID_CONTINUE_ON_ERROR: `${errorPrefix} Validation error. The continueOnError key has a value of type %s1. Specify continueOnError as boolean.`, - INVALID_UPSERT: `${errorPrefix} Validation error. The upsert key has a value of type %s1. Specify upsert as string.`, - INVALID_RETURN_TOKEN: `${errorPrefix} Validation error. The returnToken key has a value of type %s1. Specify returnToken as boolean.`, - - NO_TOKENS_WITH_TOKEN_MODE: `${errorPrefix} Validation error. Tokens weren't specified for records while 'tokenMode' was ENABLE or ENABLE_STRICT` , - INSUFFICIENT_TOKENS_PASSED_FOR_TOKEN_MODE_ENABLE_STRICT: `${errorPrefix} Validation error. 'tokenMode' is set to 'ENABLE_STRICT', but some fields are missing tokens. Specify tokens for all fields.`, - - INVALID_DOWNLOAD_URL: `${errorPrefix} Validation error. The downloadURL key has a value of type %s1. Specify downloadURL as string.`, - - EMPTY_FIELD: `${errorPrefix} Validation error. Filed value cannot be empty. Specify a valid filed value at index %s1.`, - INVALID_FIELD: `${errorPrefix} Validation error. Invalid filed value. Specify a valid filed value at index %s1 as string.`, - - INVALID_OFFSET: `${errorPrefix} Validation error. The offset key has a value of type %s1. Specify offset as string.`, - INVALID_LIMIT: `${errorPrefix} Validation error. The limit key has a value of type %s1. Specify limit as string.`, - - INVALID_ORDER_BY: `${errorPrefix} Validation error. The orderBy key has a value of type %s1. Specify orderBy as string.`, - INVALID_FIELDS: `${errorPrefix} Validation error. The fields key has a value of type %s1. Specify fields as array of strings.`, - - INVAILD_JSON_RESPONSE: `${errorPrefix} Validation error. The invalid json response. Please reach out to skyflow using requestId - %s1.`, - - EMPTY_VAULT_CLIENTS: `${errorPrefix} Validation error. No vault config found. Please add a vault config`, - EMPTY_CONNECTION_CLIENTS: `${errorPrefix} Validation error. No connection config found. Please add a connection config`, - - GENERIC_API_ERROR: `${errorPrefix} Validation error. Invalid configuration. Please add a valid vault configuration`, - INVALID_TEXT_IN_DEIDENTIFY: `${errorPrefix} Validation error. The text field is required and must be a non-empty string. Specify a valid text.`, - INVALID_ENTITIES_IN_DEIDENTIFY: `${errorPrefix} Validation error. The entities field must be an array of DetectEntities enums. Specify a valid entities.`, - INVALID_ALLOW_REGEX_LIST: `${errorPrefix} Validation error. The allowRegexList field must be an array of strings. Specify a valid allowRegexList.`, - INVALID_RESTRICT_REGEX_LIST: `${errorPrefix} Validation error. The restrictRegexList field must be an array of strings. Specify a valid restrictRegexList.`, - INVALID_TOKEN_FORMAT: `${errorPrefix} Validation error. The tokenFormat key must be an instance of TokenFormat. Specify a valid token format.`, - INVALID_TRANSFORMATIONS: `${errorPrefix} Validation error. The transformations key must be an instance of Transformations. Specify a valid transformations.`, - VAULT_TOKEN_FORMAT_NOT_ALLOWED_FOR_DEIDENTIFY_FILES: `${errorPrefix} Validation error. Vault token format is not allowed for deidentify file request.`, - - INVALID_TEXT_IN_REIDENTIFY: `${errorPrefix} Validation error. The text field is required and must be a non-empty string. Specify a valid text.`, - INVALID_REDACTED_ENTITIES_IN_REIDENTIFY: `${errorPrefix} Validation error. The redactedEntities field must be an array of DetectEntities enums. Specify a valid redactedEntities.`, - INVALID_MASKED_ENTITIES_IN_REIDENTIFY: `${errorPrefix} Validation error. The maskedEntities field must be an array of DetectEntities enums. Specify a valid maskedEntities.`, - INVALID_PLAIN_TEXT_ENTITIES_IN_REIDENTIFY: `${errorPrefix} Validation error. The plainTextEntities field must be an array of DetectEntities enums. Specify a valid plainTextEntities.`, - - INVALID_DEIDENTIFY_FILE_REQUEST: `${errorPrefix} Validation error. Invalid deidentify file request. Specify a valid deidentify file request.`, - INVALID_DEIDENTIFY_FILE_INPUT: `${errorPrefix} Validation error. Invalid deidentify file input. Please provide either a file or a file path.`, - EMPTY_FILE_OBJECT: `${errorPrefix} Validation error. File object cannot be empty. Specify a valid file object.`, - INVALID_FILE_FORMAT: `${errorPrefix} Validation error. Invalid file format. Specify a valid file format.`, - MISSING_FILE_SOURCE: `${errorPrefix} Validation error. Provide exactly one of filePath, base64, or fileObject.`, - INVALID_FILE_OBJECT: `${errorPrefix} Validation error. Invalid file object. Specify a valid file object.`, - INVALID_BASE64_STRING: `${errorPrefix} Validation error. Invalid base64 string. Specify a valid base64 string.`, - INVALID_DEIDENTIFY_FILE_OPTIONS: `${errorPrefix} Validation error. Invalid deidentify file options. Specify a valid deidentify file options.`, - INVALID_ENTITIES: `${errorPrefix} Validation error. Invalid entities. Specify valid entities as string array.`, - EMPTY_ENTITIES: `${errorPrefix} Validation error. Entities cannot be empty. Specify valid entities.`, - EMPTY_ALLOW_REGEX_LIST: `${errorPrefix} Validation error. Allow regex list cannot be empty. Specify valid allow regex list.`, - INVALID_ALLOW_REGEX: `${errorPrefix} Validation error. Invalid allow regex. Specify valid allow regex at index %s1.`, - EMPTY_RESTRICT_REGEX_LIST: `${errorPrefix} Validation error. Restrict regex list cannot be empty. Specify valid restrict regex list.`, - INVALID_RESTRICT_REGEX: `${errorPrefix} Validation error. Invalid restrict regex. Specify valid restrict regex at index %s1.`, - INVALID_OUTPUT_PROCESSED_IMAGE: `${errorPrefix} Validation error. Invalid output processed image. Specify valid output processed image as string.`, - INVALID_OUTPUT_OCR_TEXT: `${errorPrefix} Validation error. Invalid output ocr text. Specify valid output ocr text as string.`, - INVALID_MASKING_METHOD: `${errorPrefix} Validation error. Invalid masking method. Specify valid masking method as string.`, - INVALID_PIXEL_DENSITY: `${errorPrefix} Validation error. Invalid pixel density. Specify valid pixel density as string.`, - INVALID_OUTPUT_TRANSCRIPTION: `${errorPrefix} Validation error. Invalid output transcription. Specify valid output transcription as string.`, - INVALID_OUTPUT_PROCESSED_AUDIO: `${errorPrefix} Validation error. Invalid output processed audio. Specify valid output processed audio as string.`, - INVALID_MAX_RESOLUTION: `${errorPrefix} Validation error. Invalid max resolution. Specify valid max resolution as string.`, - INVALID_BLEEP: `${errorPrefix} Validation error. Invalid bleep. Specify valid bleep as object.`, - INVALID_FILE_OR_ENCODED_FILE: `${errorPrefix} . Error while decoding base64 and saving file`, - INVALID_FILE_TYPE : `${errorPrefix} Validation error. Invalid file type. Specify a valid file type.`, - INVALID_DEIDENTIFY_FILE_PATH: `${errorPrefix} Validation error. Invalid file path. Specify a valid file path.`, - INVALID_FILE_NAME: `${errorPrefix} Validation error. Invalid file name. Specify a valid file name.`, - FILE_READ_ERROR: `${errorPrefix} Validation error. Unable to read file. Verify the file path.`, - INVALID_BASE64_HEADER: `${errorPrefix} Validation error. Invalid base64 header. Specify a valid base64 header.`, - INVALID_WAIT_TIME: `${errorPrefix} Validation error. Invalid wait time. Specify a valid wait time as number and should not be greater than 20 secs.`, - INVALID_OUTPUT_DIRECTORY: `${errorPrefix} Validation error. Invalid output directory. Specify a valid output directory as string.`, - INVALID_OUTPUT_DIRECTORY_PATH: `${errorPrefix} Validation error. Invalid output directory path. Specify a valid output directory path as string.`, - EMPTY_RUN_ID: `${errorPrefix} Validation error. Run id cannot be empty. Specify a valid run id.`, - INVALID_RUN_ID: `${errorPrefix} Validation error. Invalid run id. Specify a valid run id as string.`, - INTERNAL_SERVER_ERROR: `${errorPrefix}. Internal server error. %s1.`, - INVALID_XML_FORMAT: `${errorPrefix} Validation error. Invalid XML format. Specify a valid XML format as string.`, -}; - -export default errorMessages; \ No newline at end of file diff --git a/src/vault/config/vault/index.ts b/src/vault/config/vault/index.ts deleted file mode 100644 index f33610d2..00000000 --- a/src/vault/config/vault/index.ts +++ /dev/null @@ -1,12 +0,0 @@ -// imports -import { Env } from "../../../utils"; -import Credentials from "../credentials"; - -interface VaultConfig { - vaultId: string; - clusterId: string; - env?: Env; - credentials?: Credentials; -} - -export default VaultConfig; diff --git a/src/vault/skyflow/index.ts b/src/vault/skyflow/index.ts deleted file mode 100644 index 4968a372..00000000 --- a/src/vault/skyflow/index.ts +++ /dev/null @@ -1,274 +0,0 @@ -import { CONFIG, CONTROLLER_TYPES, Env, getVaultURL, ISkyflowError, LogLevel, MessageType, parameterizedString, printLog } from "../../utils"; -import ConnectionConfig from "../config/connection"; -import VaultConfig from "../config/vault"; -import { SkyflowConfig, ClientObj } from "../types"; -import VaultController from "../controller/vault"; -import ConnectionController from "../controller/connections"; -import VaultClient from "../client"; -import Credentials from "../config/credentials"; -import SkyflowError from "../../error"; -import logs from "../../utils/logs"; -import { isLogLevel, validateConnectionConfig, validateSkyflowConfig, validateSkyflowCredentials, validateUpdateConnectionConfig, validateUpdateVaultConfig, validateVaultConfig } from "../../utils/validations"; -import SKYFLOW_ERROR_CODE from "../../error/codes"; -import DetectController from "../controller/detect"; - -class Skyflow { - - private vaultClients: ClientObj = {}; - - private connectionClients: ClientObj = {}; - - private commonCredentials?: Credentials; - - private logLevel: LogLevel = LogLevel.ERROR; - - constructor(config: SkyflowConfig) { - validateSkyflowConfig(config, this.logLevel); - printLog(logs.infoLogs.INITIALIZE_CLIENT, MessageType.LOG, this.logLevel); - if (config?.logLevel && !isLogLevel(config?.logLevel)) - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_LOG_LEVEL); - - this.logLevel = config.logLevel || LogLevel.ERROR; - printLog(parameterizedString(logs.infoLogs.CURRENT_LOG_LEVEL,[this.logLevel]), MessageType.LOG, this.logLevel); - - if (config?.skyflowCredentials) - validateSkyflowCredentials(config.skyflowCredentials) - - this.commonCredentials = config?.skyflowCredentials; - printLog(logs.infoLogs.VALIDATING_VAULT_CONFIG, MessageType.LOG, this.logLevel); - config.vaultConfigs?.map(vaultConfig => { - this.addVaultConfig(vaultConfig); - }); - printLog(logs.infoLogs.VALIDATING_CONNECTION_CONFIG, MessageType.LOG, this.logLevel); - config.connectionConfigs?.map(connectionConfig => { - this.addConnectionConfig(connectionConfig); - }); - printLog(logs.infoLogs.CLIENT_INITIALIZED, MessageType.LOG, this.logLevel); - } - - private addVaultClient(config: VaultConfig, clients: ClientObj) { - const env = config.env || Env.PROD; - const vaultUrl = getVaultURL(config.clusterId, env); - const client = new VaultClient(vaultUrl, config.vaultId, config?.credentials, this.commonCredentials, this.logLevel); - const vaultController = new VaultController(client); - const detectController = new DetectController(client); - printLog(logs.infoLogs.CONTROLLER_INITIALIZED, MessageType.LOG, this.logLevel); - clients[config.vaultId] = { config, client, vaultController, detectController }; - } - - private addConnectionClient(config: ConnectionConfig, clients: ClientObj) { - const client = new VaultClient(config.connectionUrl, '', config?.credentials, this.commonCredentials, this.logLevel); - const connectionController = new ConnectionController(client); - printLog(parameterizedString(logs.infoLogs.CONNECTION_CONTROLLER_INITIALIZED, [config.connectionId]), MessageType.LOG, this.logLevel); - clients[config.connectionId] = { config, client, connectionController }; - } - - addVaultConfig(config: VaultConfig) { - validateVaultConfig(config, this.logLevel); - this.throwErrorIfIdExits(config?.vaultId, this.vaultClients, CONFIG.VAULT_ID); - this.addVaultClient(config, this.vaultClients); - } - - addConnectionConfig(config: ConnectionConfig) { - validateConnectionConfig(config, this.logLevel); - this.throwErrorIfIdExits(config?.connectionId, this.connectionClients, CONFIG.CONNECTION_ID); - this.addConnectionClient(config, this.connectionClients); - } - - private updateVaultClient(config: VaultConfig, clients: ClientObj, idKey: string) { - const existingClient = clients[config[idKey]]; - if (existingClient) { - const updatedConfig = { ...existingClient.config, ...config }; - const vaultUrl = getVaultURL(updatedConfig.clusterId, updatedConfig.env || Env.PROD); - existingClient.config = updatedConfig; - existingClient.client.updateClientConfig(vaultUrl, updatedConfig.vaultId, updatedConfig.credentials, this.commonCredentials, this.logLevel); - } else { - this.throwErrorForUnknownId(config[idKey], idKey) - } - } - - private updateConnectionClient(config: ConnectionConfig, clients: ClientObj, idKey: string) { - const existingClient = clients[config[idKey]]; - if (existingClient) { - const updatedConfig = { ...existingClient.config, ...config }; - existingClient.config = updatedConfig; - existingClient.client.updateClientConfig(updatedConfig.connectionUrl, '', updatedConfig.credentials, this.commonCredentials, this.logLevel); - } else { - this.throwErrorForUnknownId(config[idKey], idKey) - } - } - - updateVaultConfig(config: VaultConfig) { - validateUpdateVaultConfig(config, this.logLevel); - this.updateVaultClient(config, this.vaultClients, CONFIG.VAULT_ID); - } - - updateConnectionConfig(config: ConnectionConfig) { - validateUpdateConnectionConfig(config, this.logLevel); - this.updateConnectionClient(config, this.connectionClients, CONFIG.CONNECTION_ID); - } - - getVaultConfig(vaultId: string) { - return this.getConfig(vaultId, this.vaultClients, CONFIG.VAULT_ID); - } - - removeVaultConfig(vaultId: string) { - this.removeConfig(vaultId, this.vaultClients, CONFIG.VAULT_ID); - } - - getConnectionConfig(connectionId: string) { - return this.getConfig(connectionId, this.connectionClients, CONFIG.CONNECTION_ID); - } - - removeConnectionConfig(connectionId: string) { - this.removeConfig(connectionId, this.connectionClients, CONFIG.CONNECTION_ID); - } - - private throwSkyflowError(idKey: string, errorMapping: { [key: string]: ISkyflowError }, params?: string[]) { - const errorCode = errorMapping[idKey]; - if (errorCode) { - throw new SkyflowError(errorCode, params); - } - } - - private throwErrorIfIdExits(id: string, clients: ClientObj, idKey: string) { - const errorMapping = { - [CONFIG.VAULT_ID]: SKYFLOW_ERROR_CODE.VAULT_ID_EXITS_IN_CONFIG_LIST, - [CONFIG.CONNECTION_ID]: SKYFLOW_ERROR_CODE.CONNECTION_ID_EXITS_IN_CONFIG_LIST, - }; - if(Object.keys(clients).includes(id)){ - printLog(parameterizedString(logs.infoLogs[`${idKey}_CONFIG_EXISTS`], [id]), MessageType.ERROR, this.logLevel); - this.throwSkyflowError(idKey, errorMapping, [id]); - } - } - - private throwErrorForUnknownId(id: string, idKey: string) { - const errorMapping = { - [CONFIG.VAULT_ID]: SKYFLOW_ERROR_CODE.VAULT_ID_NOT_IN_CONFIG_LIST, - [CONFIG.CONNECTION_ID]: SKYFLOW_ERROR_CODE.CONNECTION_ID_NOT_IN_CONFIG_LIST, - }; - printLog(parameterizedString(logs.infoLogs[`${idKey}_CONFIG_DOES_NOT_EXIST`], [id]), MessageType.ERROR, this.logLevel); - this.throwSkyflowError(idKey, errorMapping, [id]); - } - - private throwErrorForEmptyClients(idKey: string) { - const errorMapping = { - [CONFIG.VAULT_ID]: SKYFLOW_ERROR_CODE.EMPTY_VAULT_CLIENTS, - [CONFIG.CONNECTION_ID]: SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_CLIENTS, - }; - this.throwSkyflowError(idKey, errorMapping); - } - - private throwErrorForEmptyId(idKey: string) { - const errorMapping = { - [CONFIG.VAULT_ID]: SKYFLOW_ERROR_CODE.EMPTY_VAULT_ID_VALIDATION, - [CONFIG.CONNECTION_ID]: SKYFLOW_ERROR_CODE.EMPTY_CONNECTION_ID_VALIDATION, - }; - this.throwSkyflowError(idKey, errorMapping); - } - - private removeConfig(id: string, clients: ClientObj, idKey: string) { - if (!clients[id]) this.throwErrorForUnknownId(id, idKey); - delete clients[id]; - } - - private getConfig(id: string, clients: ClientObj, idKey: string) { - if(!id) this.throwErrorForEmptyId(idKey); - if (!clients[id]) this.throwErrorForUnknownId(id, idKey); - return clients[id].config; - } - - setLogLevel(logLevel: LogLevel) { - if (logLevel && !isLogLevel(logLevel)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_LOG_LEVEL); - } - this.logLevel = logLevel; - this.updateClients(CONFIG.LOGLEVEL); - } - - updateLogLevel(logLevel: LogLevel): Skyflow { - if (logLevel && !isLogLevel(logLevel)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_LOG_LEVEL); - } - this.logLevel = logLevel; - this.updateClients(CONFIG.LOGLEVEL); - return this; - } - - getLogLevel() { - return this.logLevel; - } - - updateSkyflowCredentials(credentials: Credentials) { - if (!credentials) - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CREDENTIALS); - validateSkyflowCredentials(credentials); - this.commonCredentials = credentials; - this.updateClients(CONFIG.CREDENTIALS); - } - - getSkyflowCredentials() { - return this.commonCredentials; - } - - vault(vaultId?: string) { - return this.getClient(vaultId, this.vaultClients, CONFIG.VAULT_ID, CONTROLLER_TYPES.VAULT) as VaultController; - } - - detect(vaultId?: string) { - return this.getClient(vaultId, this.vaultClients, CONFIG.VAULT_ID, CONTROLLER_TYPES.DETECT) as DetectController; - } - - connection(connectionId?: string) { - return this.getClient(connectionId, this.connectionClients, CONFIG.CONNECTION_ID, CONTROLLER_TYPES.CONNECTION) as ConnectionController; - } - - private getClient( - id: string | undefined, - clients: ClientObj, - idKey: string, - controllerType: string - ) { - if (Object.keys(clients).length === 0) { - this.throwErrorForEmptyClients(idKey); - } - - const clientId = id || Object.keys(clients)[0]; - if (clientId) { - const clientData = clients[clientId]; - - if (controllerType === CONTROLLER_TYPES.VAULT && clientData?.vaultController) { - return clientData.vaultController; - } - - if (controllerType === CONTROLLER_TYPES.DETECT && clientData?.detectController) { - return clientData.detectController; - } - - if (controllerType === CONTROLLER_TYPES.CONNECTION && clientData?.connectionController) { - return clientData.connectionController; - } - - this.throwErrorForUnknownId(clientId, idKey); - } - throw new SkyflowError(SKYFLOW_ERROR_CODE.INTERNAL_SERVER_ERROR); - } - - private updateClients(updateType: string) { - this.updateClient(updateType, this.vaultClients); - this.updateClient(updateType, this.connectionClients); - } - - private updateClient(updateType: string, list: ClientObj) { - Object.values(list).forEach(clientConfig => { - if (updateType === CONFIG.LOGLEVEL) { - clientConfig.client.setLogLevel(this.logLevel); - } else if (updateType === CONFIG.CREDENTIALS) { - clientConfig.client.updateSkyflowCredentials(this.commonCredentials); - } - }); - } - -} - -export default Skyflow; diff --git a/tsconfig.json b/tsconfig.json index dbd34688..5cfce407 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -1,26 +1,8 @@ { - "compilerOptions": { - "sourceMap": true, - "noImplicitAny": false, - "target": "es6", - "lib": ["es2015", "es2017", "es2019", "dom"], - "esModuleInterop": true, - "module": "commonjs", - "moduleResolution": "node", - "removeComments": false, - "allowSyntheticDefaultImports": true, - "allowJs": true, - "strict": true, - "skipLibCheck": true, - "forceConsistentCasingInFileNames": true, - "resolveJsonModule": false, - "isolatedModules": false, - "noEmit": false, - "declaration": true, - "rootDir": "src", - "outDir": "lib", - "types": ["node"] - }, - "include": ["src/**/*.ts","typings.d.ts"], - "exclude": ["node_modules", "dist", "tests/**/*.test.*"], + "files": [], + "references": [ + { "path": "./common" }, + { "path": "./skyvault" }, + { "path": "./flowvault" } + ] } From d15f464aba7cebaf81d826657be8acfa99e559eb Mon Sep 17 00:00:00 2001 From: saileshwar-skyflow Date: Fri, 23 Oct 2026 15:43:18 +0530 Subject: [PATCH 02/11] =?UTF-8?q?SK-3225:=20internal=20release=20=E2=80=94?= =?UTF-8?q?=20don't=20push=20version=20bump=20back=20to=20branch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The internal (JFrog) build's 'Commit Changes' step force-pushed the version bump to the release branch, which failed with a permission denial (the PAT_ACTIONS owner lacks push access) and killed the job before Publish. The push-back is also unnecessary — Publish uses the already-bumped working tree — and harmful, since src/version.ts is not path-ignored so the push would re-trigger this workflow in a loop. Restrict the version-bump commit to public/beta releases; internal builds now bump, build, and publish to JFrog without mutating the branch. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/common-release.yml | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/.github/workflows/common-release.yml b/.github/workflows/common-release.yml index c220c487..3c058dea 100644 --- a/.github/workflows/common-release.yml +++ b/.github/workflows/common-release.yml @@ -76,21 +76,20 @@ jobs: fi echo "branch_name=$BRANCH_NAME" >> $GITHUB_ENV + # Only public/beta record the released version back on the source branch. + # Internal (JFrog) builds publish the bumped working tree directly and do + # NOT push back: the branch needs no mutation, pushing would need write + # access to the release branch, and src/version.ts is not path-ignored so + # a push-back would re-trigger this workflow in a loop. - name: Commit Changes + if: ${{ inputs.release-type == 'beta' || inputs.release-type == 'public' }} run: | git config user.name ${{ github.actor }} git config user.email ${{ github.actor }}@users.noreply.github.com - if [[ "${{ inputs.release-type }}" == "beta" || "${{ inputs.release-type }}" == "public" ]]; then - git checkout ${{ env.branch_name }} - fi + git checkout ${{ env.branch_name }} git add ${{ inputs.package-dir }}/package.json ${{ inputs.package-dir }}/src/version.ts - if [[ "${{ inputs.release-type }}" == "beta" || "${{ inputs.release-type }}" == "public" ]]; then - git commit -m "[AUTOMATED] Release ${{ inputs.product }} - ${{ steps.version.outputs.value }}" || echo "nothing to commit" - git push origin ${{ env.branch_name }} - elif [[ "${{ inputs.release-type }}" == "internal" ]]; then - git commit -m "[AUTOMATED] Private Release ${{ inputs.product }} ${{ steps.version.outputs.value }}-dev.$(git rev-parse --short $GITHUB_SHA)" || echo "nothing to commit" - git push origin ${{ github.ref_name }} -f - fi + git commit -m "[AUTOMATED] Release ${{ inputs.product }} - ${{ steps.version.outputs.value }}" || echo "nothing to commit" + git push origin ${{ env.branch_name }} - name: Publish working-directory: ${{ inputs.package-dir }} From 34b8dc8c065d10fc498a3b52507c126fba4fea97 Mon Sep 17 00:00:00 2001 From: saileshwar-skyflow Date: Fri, 23 Oct 2026 15:47:05 +0530 Subject: [PATCH 03/11] =?UTF-8?q?SK-3225:=20internal=20publish=20=E2=80=94?= =?UTF-8?q?=20set=20JFrog=20registry=20via=20.npmrc,=20not=20npm=20config?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit npm config set cannot run from inside a workspace package directory: npm detects the implicit workspace and aborts with ENOWORKSPACES, which killed the internal Publish step. Write the JFrog registry + auth to a local .npmrc in the package dir instead (npm publish itself supports workspaces and runs fine). Co-Authored-By: Claude Opus 4.8 --- .github/workflows/common-release.yml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/common-release.yml b/.github/workflows/common-release.yml index 3c058dea..bceead17 100644 --- a/.github/workflows/common-release.yml +++ b/.github/workflows/common-release.yml @@ -99,9 +99,14 @@ jobs: elif [[ "${{ inputs.release-type }}" == "public" ]]; then npm publish elif [[ "${{ inputs.release-type }}" == "internal" ]]; then + # Point npm at the JFrog registry with auth via a local .npmrc. + # `npm config set` cannot run from inside a workspace package dir + # (npm errors with ENOWORKSPACES), so write the config file directly. JFROG_AUTH=$(echo -n "$JFROG_USERNAME:$JFROG_PASSWORD" | base64 -w 0) - npm config set registry https://prekarilabs.jfrog.io/prekarilabs/api/npm/npm/ - npm config set //prekarilabs.jfrog.io/prekarilabs/api/npm/npm/:_auth "$JFROG_AUTH" + { + echo "registry=https://prekarilabs.jfrog.io/prekarilabs/api/npm/npm/" + echo "//prekarilabs.jfrog.io/prekarilabs/api/npm/npm/:_auth=${JFROG_AUTH}" + } > .npmrc npm publish fi env: From 1f545e4c1eb50be190b19a750075576efdb36d5c Mon Sep 17 00:00:00 2001 From: saileshwar-skyflow Date: Fri, 23 Oct 2026 15:50:02 +0530 Subject: [PATCH 04/11] =?UTF-8?q?SK-3225:=20internal=20publish=20=E2=80=94?= =?UTF-8?q?=20force=20JFrog=20registry=20via=20--registry=20flag?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A workspace member's own .npmrc is ignored by npm (it read the root config and defaulted to registry.npmjs.org, so publish 404'd on public npm). Pass the JFrog registry with --registry on the publish command (highest precedence) and put its auth in the workspace-root .npmrc that npm actually reads. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/common-release.yml | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/.github/workflows/common-release.yml b/.github/workflows/common-release.yml index bceead17..bfdc9e74 100644 --- a/.github/workflows/common-release.yml +++ b/.github/workflows/common-release.yml @@ -99,15 +99,13 @@ jobs: elif [[ "${{ inputs.release-type }}" == "public" ]]; then npm publish elif [[ "${{ inputs.release-type }}" == "internal" ]]; then - # Point npm at the JFrog registry with auth via a local .npmrc. - # `npm config set` cannot run from inside a workspace package dir - # (npm errors with ENOWORKSPACES), so write the config file directly. + # Publish to the internal JFrog registry. npm ignores a workspace + # member's own .npmrc and `npm config set` can't run inside a + # workspace dir (ENOWORKSPACES), so pass the registry on the command + # line and add its auth to the workspace-root .npmrc that npm reads. JFROG_AUTH=$(echo -n "$JFROG_USERNAME:$JFROG_PASSWORD" | base64 -w 0) - { - echo "registry=https://prekarilabs.jfrog.io/prekarilabs/api/npm/npm/" - echo "//prekarilabs.jfrog.io/prekarilabs/api/npm/npm/:_auth=${JFROG_AUTH}" - } > .npmrc - npm publish + echo "//prekarilabs.jfrog.io/prekarilabs/api/npm/npm/:_auth=${JFROG_AUTH}" >> "$GITHUB_WORKSPACE/.npmrc" + npm publish --registry https://prekarilabs.jfrog.io/prekarilabs/api/npm/npm/ fi env: NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} From a6a412dddb00ceb8d9939c5521230a4824f905e6 Mon Sep 17 00:00:00 2001 From: saileshwar-skyflow Date: Wed, 28 Oct 2026 15:23:31 +0530 Subject: [PATCH 05/11] flowvault: gitignore local-only samples-local runner samples-local/ holds real creds in .env and is for manual local testing; it must never be committed. Co-Authored-By: Claude Opus 4.8 --- flowvault/.gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/flowvault/.gitignore b/flowvault/.gitignore index 80e178c4..f865b72f 100644 --- a/flowvault/.gitignore +++ b/flowvault/.gitignore @@ -1,3 +1,6 @@ # npm pack artifacts (local) *.tgz + +# Local-only sample runner (real creds in .env — never commit) +samples-local/ From 6d8926670d7356469dc11717e6015c6a7bd0c8ee Mon Sep 17 00:00:00 2001 From: saileshwar-skyflow Date: Wed, 28 Oct 2026 15:23:38 +0530 Subject: [PATCH 06/11] flowvault: typed Token response model + query skyflowId normalization - Add a typed Token { token?, tokenGroupName?, path? } and surface response tokens as Record on insert/get/update, parsing the wire token map (list / bare value / multiple groups) the same way the Java and Python flowvault SDKs do. Export Token; regenerate the api-report snapshot. - Normalize the query data plane's system key skyflowID -> skyflowId in rows and metadata.columns, so query matches every other operation's response. - Update unit tests and the README accordingly. Co-Authored-By: Claude Opus 4.8 --- flowvault/README.md | 2 + .../api-report/skyflow-flowvault-node.api.md | 16 ++++- flowvault/src/controller/vault/index.ts | 69 +++++++++++++++++-- flowvault/src/index.ts | 1 + flowvault/src/types/index.ts | 19 ++++- flowvault/test/controller/get.test.ts | 4 +- flowvault/test/controller/insert.test.ts | 33 ++++++++- flowvault/test/controller/query.test.ts | 13 ++++ flowvault/test/controller/update.test.ts | 4 +- flowvault/test/smoke.test.ts | 4 +- 10 files changed, 146 insertions(+), 19 deletions(-) diff --git a/flowvault/README.md b/flowvault/README.md index 323a9adf..e63671e4 100644 --- a/flowvault/README.md +++ b/flowvault/README.md @@ -157,6 +157,8 @@ const options = new InsertOptions(); const res = await vault.insert(req, options); // res.records[i]: { tableName, skyflowId, tokens, data, hashedData, httpCode, error, requestId } +// tokens is keyed by column: Record, where Token is +// { token, tokenGroupName, path? } — same shape as the Java/Python SDKs. ``` ### Get diff --git a/flowvault/api-report/skyflow-flowvault-node.api.md b/flowvault/api-report/skyflow-flowvault-node.api.md index b97f01fc..4d356193 100644 --- a/flowvault/api-report/skyflow-flowvault-node.api.md +++ b/flowvault/api-report/skyflow-flowvault-node.api.md @@ -282,7 +282,7 @@ export interface GetResponseRecord { // (undocumented) tableName?: string | null; // (undocumented) - tokens?: Record | null; + tokens?: Record | null; } // @public @@ -388,7 +388,7 @@ export interface InsertResponseRecord { // (undocumented) tableName?: string | null; // (undocumented) - tokens?: Record | null; + tokens?: Record | null; } // @public @@ -457,6 +457,16 @@ class Skyflow extends BaseSkyflow { export { Skyflow } export default Skyflow; +// @public +export interface Token { + // (undocumented) + path?: string; + // (undocumented) + token?: string; + // (undocumented) + tokenGroupName?: string; +} + // @public export interface TokenGroupRedaction { // (undocumented) @@ -522,7 +532,7 @@ export interface UpdateResponseRecord { // (undocumented) tableName?: string | null; // (undocumented) - tokens?: Record | null; + tokens?: Record | null; } // @public diff --git a/flowvault/src/controller/vault/index.ts b/flowvault/src/controller/vault/index.ts index f1f2337e..ba321fac 100644 --- a/flowvault/src/controller/vault/index.ts +++ b/flowvault/src/controller/vault/index.ts @@ -53,6 +53,7 @@ import { UploadFilesResponseRecord, DeleteFilesResponseRecord, UpsertOptions, + Token, } from '../../types'; import { validateInsertRequest, @@ -73,6 +74,55 @@ interface PreparedColumn { contentType: string; } +/** The data plane names the system primary-key column `skyflowID` on the wire; + * the SDK surfaces it as `skyflowId` everywhere else. */ +const WIRE_SKYFLOW_ID = 'skyflowID'; +const SKYFLOW_ID = 'skyflowId'; + +/** Parse one wire token entry into a typed {@link Token}. Mirrors the Java/Python + * flowvault `toToken`: a map yields token/tokenGroupName/path; a bare value + * (a column tokenized against a single unnamed group) yields a token-only entry. */ +function toToken(entry: unknown): Token | null { + if (entry !== null && typeof entry === 'object' && !Array.isArray(entry)) { + const map = entry as Record; + const token: Token = {}; + if (map.token != null) token.token = String(map.token); + if (map.tokenGroupName != null) token.tokenGroupName = String(map.tokenGroupName); + if (map.path != null) token.path = String(map.path); + return token; + } + if (entry != null) return { token: String(entry) }; + return null; +} + +/** Parse a single column's wire value (a list of entries, or one entry) into Token[]. */ +function parseTokenEntries(rawValue: unknown): Token[] | null { + if (rawValue == null) return null; + const parsed: Token[] = []; + const entries = Array.isArray(rawValue) ? rawValue : [rawValue]; + for (const entry of entries) { + const token = toToken(entry); + if (token) parsed.push(token); + } + return parsed; +} + +/** + * Parse the generic wire token map (`column -> value`) into `column -> Token[]`. + * Mirrors `Token.parseTokens` in the Java/Python flowvault SDKs so insert/get/ + * update responses surface typed tokens instead of untyped maps. Returns null + * when there are no tokens. + */ +function parseTokens(raw: unknown): Record | null { + if (raw == null || typeof raw !== 'object') return null; + const parsed: Record = {}; + for (const [column, value] of Object.entries(raw as Record)) { + const tokens = parseTokenEntries(value); + if (tokens != null) parsed[column] = tokens; + } + return parsed; +} + /** * Operation names handed to a request interceptor via {@link RequestContext}. * Kept identical to the Python/Java FlowVault SDKs so interceptor logic ports @@ -263,7 +313,7 @@ class VaultController { const result: InsertResponseRecord = { tableName: record.tableName ?? null, skyflowId: record.skyflowID ?? null, - tokens: record.tokens ?? null, + tokens: parseTokens(record.tokens), data: record.data ?? null, hashedData: record.hashedData ?? null, httpCode: record.httpCode, @@ -349,7 +399,7 @@ class VaultController { tableName: record.tableName ?? null, data: record.data ?? null, hashedData: record.hashedData ?? null, - tokens: record.tokens ?? null, + tokens: parseTokens(record.tokens), httpCode: record.httpCode, error: hasError ? record.error : null, }; @@ -414,7 +464,7 @@ class VaultController { const result: UpdateResponseRecord = { skyflowId: record.skyflowID ?? null, tableName: record.tableName ?? null, - tokens: record.tokens ?? null, + tokens: parseTokens(record.tokens), data: record.data ?? null, hashedData: record.hashedData ?? null, httpCode: record.httpCode, @@ -500,12 +550,21 @@ class VaultController { const requestId = this.getRequestId(rawResponse) ?? ''; printLog(logs.infoLogs.QUERY_SUCCESS, MessageType.LOG, this.logLevel); - const fields = (data.records ?? []).map((record) => record.data ?? {}); + // The query data plane returns the system primary-key column as + // `skyflowID`; normalize it to `skyflowId` so query rows (and the column + // list) use the same key as every other operation's response records. + const normalizeKey = (key: string): string => (key === WIRE_SKYFLOW_ID ? SKYFLOW_ID : key); + const fields = (data.records ?? []).map((record) => { + const row = record.data ?? {}; + const normalized: Record = {}; + for (const [key, value] of Object.entries(row)) normalized[normalizeKey(key)] = value; + return normalized; + }); // NOTE: generated ExecuteQueryResponse has no per-record errors block. return new QueryResponse({ fields, errors: null, - metadata: { columns: data.metadata?.columns ?? [] }, + metadata: { columns: (data.metadata?.columns ?? []).map(normalizeKey) }, requestId, }); } catch (error) { diff --git a/flowvault/src/index.ts b/flowvault/src/index.ts index bfea3a6b..a6756510 100644 --- a/flowvault/src/index.ts +++ b/flowvault/src/index.ts @@ -48,6 +48,7 @@ export { default as DeleteFilesResponse } from './model/response/delete-files'; // Record / helper types. export type { GetRecord } from './model/request/get'; export type { + Token, InsertRecord, UpsertOptions, ColumnRedaction, diff --git a/flowvault/src/types/index.ts b/flowvault/src/types/index.ts index dc27da08..b2dea8e0 100644 --- a/flowvault/src/types/index.ts +++ b/flowvault/src/types/index.ts @@ -76,6 +76,19 @@ export interface DeleteFilesRecord { /** Result of uploading a file to a single column. */ export type UploadStatus = 'UPLOADED' | 'FAILED' | 'SKIPPED'; +/** + * A single parsed token, surfaced per column in insert/get/update responses. + * Mirrors the `Token` type in the Java/Python flowvault SDKs. The SDK parses the + * generic wire token map into `Record`. `path` is populated only + * when the vault returns a token path; a column tokenized against a single + * unnamed group can come back as a bare value (then only `token` is set). + */ +export interface Token { + token?: string; + tokenGroupName?: string; + path?: string; +} + // --------------------------------------------------------------------------- // Unified response records. Every record carries httpCode and an optional // error. requestId is populated only when error is present. @@ -86,7 +99,7 @@ export interface InsertResponseRecord { skyflowId?: string | null; // tokens and data are surfaced separately (matches the Java/Python flowvault // SDKs); either may be present depending on the vault configuration. - tokens?: Record | null; + tokens?: Record | null; data?: Record | null; hashedData?: Record | null; httpCode: number; @@ -99,7 +112,7 @@ export interface GetResponseRecord { tableName?: string | null; data?: Record | null; hashedData?: Record | null; - tokens?: Record | null; + tokens?: Record | null; httpCode: number; error?: string | null; requestId?: string | null; @@ -108,7 +121,7 @@ export interface GetResponseRecord { export interface UpdateResponseRecord { skyflowId?: string | null; tableName?: string | null; - tokens?: Record | null; + tokens?: Record | null; data?: Record | null; hashedData?: Record | null; httpCode: number; diff --git a/flowvault/test/controller/get.test.ts b/flowvault/test/controller/get.test.ts index 9b276fa4..e27d267a 100644 --- a/flowvault/test/controller/get.test.ts +++ b/flowvault/test/controller/get.test.ts @@ -13,7 +13,7 @@ describe('VaultController.get', () => { tableName: 'cards', data: { name: 'alice' }, hashedData: { name: 'h' }, - tokens: { name: 'tok' }, + tokens: { name: [{ token: 'tok', tokenGroupName: 'deterministic_string' }] }, httpCode: 200, }, ], @@ -34,7 +34,7 @@ describe('VaultController.get', () => { expect(record.skyflowId).toBe('id1'); expect(record.data).toEqual({ name: 'alice' }); expect(record.hashedData).toEqual({ name: 'h' }); - expect(record.tokens).toEqual({ name: 'tok' }); + expect(record.tokens).toEqual({ name: [{ token: 'tok', tokenGroupName: 'deterministic_string' }] }); expect(record.httpCode).toBe(200); expect(record.error).toBeNull(); expect(record.requestId).toBeUndefined(); diff --git a/flowvault/test/controller/insert.test.ts b/flowvault/test/controller/insert.test.ts index 4f004b21..b657b41e 100644 --- a/flowvault/test/controller/insert.test.ts +++ b/flowvault/test/controller/insert.test.ts @@ -17,7 +17,7 @@ describe('VaultController.insert', () => { records: [ { skyflowID: 'id1', - tokens: { card_number: 'tok_1' }, + tokens: { card_number: [{ token: 'tok_1', tokenGroupName: 'deterministic_string', path: 'p1' }] }, hashedData: { card_number: 'hash_1' }, httpCode: 200, tableName: 'cards', @@ -46,7 +46,8 @@ describe('VaultController.insert', () => { // Response mapped field-by-field. const record = resp.records[0]; expect(record.skyflowId).toBe('id1'); - expect(record.tokens).toEqual({ card_number: 'tok_1' }); + // tokens are parsed into typed Token[] per column (path preserved when present). + expect(record.tokens).toEqual({ card_number: [{ token: 'tok_1', tokenGroupName: 'deterministic_string', path: 'p1' }] }); expect(record.data).toBeNull(); expect(record.hashedData).toEqual({ card_number: 'hash_1' }); expect(record.tableName).toBe('cards'); @@ -71,6 +72,34 @@ describe('VaultController.insert', () => { expect(resp.records[0].tokens).toBeNull(); }); + it('parses token wire shapes: list of maps, bare value, and multiple groups per column', async () => { + const insertRecords = resourceMethod({ + records: [ + { + skyflowID: 'id1', + httpCode: 200, + tokens: { + // list of token maps (the common case) + ssn: [{ token: 't1', tokenGroupName: 'g1', path: 'p1' }], + // a bare value -> token-only entry, wrapped in a one-element list (matches Java) + email: 'bare-tok', + // two groups for one column + name: [{ token: 'a', tokenGroupName: 'g1' }, { token: 'b', tokenGroupName: 'g2' }], + }, + }, + ], + }); + const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); + + const resp = await controller.insert(new InsertRequest('cards', [{ data: { ssn: '1' } }])); + + expect(resp.records[0].tokens).toEqual({ + ssn: [{ token: 't1', tokenGroupName: 'g1', path: 'p1' }], + email: [{ token: 'bare-tok' }], + name: [{ token: 'a', tokenGroupName: 'g1' }, { token: 'b', tokenGroupName: 'g2' }], + }); + }); + it('sends a per-record tableName when records carry their own table (multi-table insert)', async () => { const insertRecords = resourceMethod({ records: [] }); const controller = new VaultController(makeFakeClient({ records: { insertRecords } })); diff --git a/flowvault/test/controller/query.test.ts b/flowvault/test/controller/query.test.ts index e9ce3e95..bb5c814a 100644 --- a/flowvault/test/controller/query.test.ts +++ b/flowvault/test/controller/query.test.ts @@ -33,6 +33,19 @@ describe('VaultController.query', () => { expect(resp.requestId).toBe(REQUEST_ID); }); + it('normalizes the data plane skyflowID key to skyflowId in rows and columns', async () => { + const executeQuery = resourceMethod({ + records: [{ data: { skyflowID: 'id-1', name: 'alice' } }], + metadata: { columns: ['skyflowID', 'name'] }, + }); + const controller = new VaultController(makeFakeClient({ query: { executeQuery } })); + + const resp = await controller.query(new QueryRequest('SELECT * FROM cards')); + + expect(resp.fields).toEqual([{ skyflowId: 'id-1', name: 'alice' }]); + expect(resp.metadata.columns).toEqual(['skyflowId', 'name']); + }); + it('defaults fields to {} and columns to [] when absent; requestId falls back to ""', async () => { const executeQuery = resourceMethod({ records: [{}, { data: null }] }, rawResponseNoId); const controller = new VaultController(makeFakeClient({ query: { executeQuery } })); diff --git a/flowvault/test/controller/update.test.ts b/flowvault/test/controller/update.test.ts index f94e9bd3..f1f3349d 100644 --- a/flowvault/test/controller/update.test.ts +++ b/flowvault/test/controller/update.test.ts @@ -14,7 +14,7 @@ silenceConsole(); describe('VaultController.update', () => { it('builds skyflowID/data(+BYOT tokens) records and maps the response', async () => { const updateRecords = resourceMethod({ - records: [{ skyflowID: 'id1', tokens: { x: 'tok' }, hashedData: { x: 'h' }, tableName: 'cards', httpCode: 200 }], + records: [{ skyflowID: 'id1', tokens: { x: [{ token: 'tok', tokenGroupName: 'deterministic_string' }] }, hashedData: { x: 'h' }, tableName: 'cards', httpCode: 200 }], }); const controller = new VaultController(makeFakeClient({ records: { updateRecords } })); @@ -30,7 +30,7 @@ describe('VaultController.update', () => { const record = resp.records[0]; expect(record.skyflowId).toBe('id1'); - expect(record.tokens).toEqual({ x: 'tok' }); + expect(record.tokens).toEqual({ x: [{ token: 'tok', tokenGroupName: 'deterministic_string' }] }); expect(record.data).toBeNull(); expect(record.hashedData).toEqual({ x: 'h' }); expect(record.tableName).toBe('cards'); diff --git a/flowvault/test/smoke.test.ts b/flowvault/test/smoke.test.ts index 89a5e35f..8a64f964 100644 --- a/flowvault/test/smoke.test.ts +++ b/flowvault/test/smoke.test.ts @@ -114,7 +114,7 @@ describe('Skyflow facade', () => { describe('VaultController maps generated responses', () => { it('insert maps records to InsertResponse', async () => { const insertRecords = resourceMethod({ - records: [{ skyflowID: 'id1', tokens: { card_number: 'tok_1' }, httpCode: 200, tableName: 'cards' }], + records: [{ skyflowID: 'id1', tokens: { card_number: [{ token: 'tok_1', tokenGroupName: 'deterministic_string' }] }, httpCode: 200, tableName: 'cards' }], }); const controller = new VaultController( makeFakeClient({ records: { insertRecords } }) as any, @@ -124,7 +124,7 @@ describe('VaultController maps generated responses', () => { expect(resp).toBeInstanceOf(InsertResponse); expect(insertRecords).toHaveBeenCalledTimes(1); expect(resp.records[0].skyflowId).toBe('id1'); - expect(resp.records[0].tokens).toEqual({ card_number: 'tok_1' }); + expect(resp.records[0].tokens).toEqual({ card_number: [{ token: 'tok_1', tokenGroupName: 'deterministic_string' }] }); expect(resp.records[0].tableName).toBe('cards'); expect(resp.records[0].httpCode).toBe(200); expect(resp.records[0].error).toBeNull(); From eb3ab59b3b7b9b93cd0a20f500e70128eec473a3 Mon Sep 17 00:00:00 2001 From: skyflow-bharti Date: Fri, 6 Nov 2026 09:40:33 +0530 Subject: [PATCH 07/11] SK-3076 added detect support --- README.md | 388 +-- common/src/_generated_/rest/Client.ts | 37 +- .../rest/api/errors/BadRequestError.ts | 3 +- .../rest/api/errors/InternalServerError.ts | 19 + .../rest/api/errors/NotFoundError.ts | 3 +- .../rest/api/errors/UnauthorizedError.ts | 3 +- .../src/_generated_/rest/api/errors/index.ts | 1 + common/src/_generated_/rest/api/index.ts | 2 +- .../resources/authentication/client/Client.ts | 51 +- .../client/requests/GetAuthTokenRequest.ts | 25 + .../client/requests/V1GetAuthTokenRequest.ts | 25 - .../authentication/client/requests/index.ts | 2 +- .../api/resources/filesV2/client/Client.ts | 412 +++ .../api/resources/filesV2/client/index.ts | 1 + .../requests/DeidentifyFileRequestV2.ts | 71 + .../client/requests/GetRunV2Request.ts | 13 + .../requests/ReidentifyFileRequestV2.ts | 43 + .../filesV2/client/requests/index.ts | 3 + .../rest/api/resources/filesV2/index.ts | 2 + .../DeidentifyFileRequestV2DataFormat.ts | 56 + .../DeidentifyFileRequestV2DataSource.ts | 13 + .../ReidentifyFileRequestV2DataFormat.ts | 18 + .../ReidentifyFileRequestV2DataSource.ts | 13 + .../rest/api/resources/filesV2/types/index.ts | 4 + .../resources/guardrailsV2/client/Client.ts | 144 ++ .../resources/guardrailsV2/client/index.ts | 1 + .../requests/DetectGuardrailsRequestV2.ts | 23 + .../guardrailsV2/client/requests/index.ts | 1 + .../rest/api/resources/guardrailsV2/index.ts | 1 + .../_generated_/rest/api/resources/index.ts | 9 + .../api/resources/stringsV2/client/Client.ts | 293 +++ .../api/resources/stringsV2/client/index.ts | 1 + .../requests/DeidentifyStringRequestV2.ts | 54 + .../requests/ReidentifyStringRequestV2.ts | 35 + .../stringsV2/client/requests/index.ts | 2 + .../rest/api/resources/stringsV2/index.ts | 1 + .../resources/tokenExchange/client/Client.ts | 150 ++ .../resources/tokenExchange/client/index.ts | 1 + .../client/requests/GetStsTokenRequest.ts | 21 + .../tokenExchange/client/requests/index.ts | 1 + .../rest/api/resources/tokenExchange/index.ts | 1 + .../src/_generated_/rest/api/types/AudioV2.ts | 16 + .../rest/api/types/AudioV2Bleep.ts | 17 + .../api/types/AudioV2OutputTranscription.ts | 16 + .../api/types/DeidentifyFileResponseV2.ts | 11 + .../api/types/DeidentifyStringResponseV2.ts | 16 + ...eidentifyStringResponseV2DetectedEntity.ts | 20 + ...eidentifyStringResponseV2EntityLocation.ts | 17 + .../src/_generated_/rest/api/types/Detect.ts | 23 + .../rest/api/types/DetectConfigV2.ts | 27 + .../rest/api/types/DetectConfigV2VaultType.ts | 14 + .../rest/api/types/DetectCustomTypes.ts | 23 + .../DetectCustomTypesDeidentificationType.ts | 16 + .../types/DetectCustomTypesIncludePrefix.ts | 12 + .../rest/api/types/DetectEntity.ts | 22 + .../types/DetectEntityDeidentificationType.ts | 16 + .../rest/api/types/DetectEntityEntityType.ts | 152 ++ .../api/types/DetectEntityIncludePrefix.ts | 12 + .../api/types/DetectGuardrailsResponseV2.ts | 19 + .../DetectGuardrailsResponseV2Validation.ts | 12 + .../rest/api/types/DetectObjectEntity.ts | 15 + .../DetectObjectEntityDeidentificationType.ts | 13 + .../api/types/DetectObjectEntityEntityType.ts | 16 + .../rest/api/types/DetectReturnEntities.ts | 13 + .../rest/api/types/DetectRunsResponseV2.ts | 20 + .../types/DetectRunsResponseV2OutputType.ts | 13 + .../api/types/DetectRunsResponseV2Status.ts | 15 + .../_generated_/rest/api/types/DocumentV2.ts | 12 + .../rest/api/types/DocumentV2Pdf.ts | 17 + .../api/types/DocumentV2PdfProcessingMode.ts | 13 + .../rest/api/types/EntityTransformation.ts | 12 + .../{GooglerpcStatus.ts => ErrorResponse.ts} | 6 +- .../rest/api/types/ErrorResponseError.ts | 14 + .../_generated_/rest/api/types/FileMapping.ts | 19 + .../_generated_/rest/api/types/FileOutput.ts | 17 + .../types/FileOutputProcessedFileExtension.ts | 56 + .../api/types/FileOutputProcessedFileType.ts | 26 + ...kenResponse.ts => GetAuthTokenResponse.ts} | 2 +- .../rest/api/types/GetStsTokenResponse.ts | 12 + .../_generated_/rest/api/types/HttpCode.ts | 8 + .../src/_generated_/rest/api/types/ImageV2.ts | 17 + .../rest/api/types/ImageV2MaskingMethod.ts | 13 + .../src/_generated_/rest/api/types/Media.ts | 14 + .../src/_generated_/rest/api/types/Metrics.ts | 21 + .../_generated_/rest/api/types/ProtobufAny.ts | 7 - .../rest/api/types/RedactionLevel.ts | 19 + .../api/types/RedactionLevelEntityName.ts | 152 ++ .../api/types/RedactionLevelRedactionType.ts | 14 + .../api/types/ReidentifyFileResponseV2.ts | 18 + .../ReidentifyFileResponseV2OutputType.ts | 13 + .../types/ReidentifyFileResponseV2Status.ts | 15 + .../api/types/ReidentifyStringResponseV2.ts | 14 + .../api/types/TransformationShiftDates.ts | 13 + .../src/_generated_/rest/api/types/index.ts | 54 +- common/src/_generated_/rest/environments.ts | 4 +- common/src/detect/controller/index.ts | 333 +++ common/src/detect/index.ts | 27 + .../detect/internal/deidentify-file-flow.ts | 37 + common/src/detect/internal/errors.ts | 43 + .../src/detect/internal/file-output-writer.ts | 114 + common/src/detect/internal/request-mapper.ts | 339 +++ common/src/detect/internal/response-mapper.ts | 182 ++ common/src/detect/internal/run-poller.ts | 95 + common/src/detect/internal/validations.ts | 165 ++ .../model/request/check-guardrails/index.ts | 49 + .../model/request/deidentify-file/index.ts | 142 ++ .../model/request/deidentify-string/index.ts | 57 + .../src/detect/model/request/get-run/index.ts | 19 + .../model/request/reidentify-file/index.ts | 66 + .../model/request/reidentify-string/index.ts | 38 + .../model/response/check-guardrails/index.ts | 30 + .../model/response/deidentify-file/index.ts | 31 + .../model/response/deidentify-string/index.ts | 21 + .../detect/model/response/get-run/index.ts | 43 + .../model/response/reidentify-file/index.ts | 30 + .../model/response/reidentify-string/index.ts | 17 + common/src/detect/types.ts | 407 +++ common/src/error/codes/index.ts | 61 +- common/src/error/messages/index.ts | 66 +- common/src/index.ts | 13 +- common/src/service-account/index.ts | 10 +- common/src/types/index.ts | 17 - common/src/utils/index.ts | 135 - common/src/utils/logs/index.ts | 84 +- common/test/detect/controller.test.js | 315 +++ common/test/detect/edge-cases.test.js | 215 ++ common/test/detect/file-output-writer.test.js | 80 + common/test/detect/models.test.js | 123 + common/test/detect/request-mapper.test.js | 254 ++ common/test/detect/response-mapper.test.js | 150 ++ common/test/detect/run-poller.test.js | 91 + common/test/detect/test-types.js | 29 + common/test/detect/validations.test.js | 252 ++ common/test/service-account/token.test.js | 18 +- docs/api_reference.md | 444 ++-- docs/sdk-split-plan.md | 22 +- flowvault/README.md | 294 ++- .../api-report/skyflow-flowvault-node.api.md | 5 + flowvault/samples/README.md | 12 + .../samples/detect-api/check-guardrails.ts | 82 + .../detect-api/deidentify-file-async.ts | 126 + .../samples/detect-api/deidentify-file.ts | 92 + .../samples/detect-api/deidentify-string.ts | 109 + flowvault/samples/detect-api/get-run.ts | 90 + .../samples/detect-api/reidentify-file.ts | 93 + .../samples/detect-api/reidentify-string.ts | 84 + flowvault/src/client/index.ts | 9 + flowvault/src/controller/detect/index.ts | 33 + flowvault/src/detect/types.ts | 35 + flowvault/src/index.ts | 5 + flowvault/src/skyflow/index.ts | 15 +- flowvault/test/controller/detect.test.ts | 100 + samples/README.md | 15 +- samples/detect-api/check-guardrails.ts | 82 + samples/detect-api/deidentify-file-async.ts | 126 + .../deidentify-file-with-filepath-async.ts | 145 -- .../deidentify-file-with-filepath.ts | 150 -- samples/detect-api/deidentify-file.ts | 221 +- samples/detect-api/deidentify-string.ts | 109 + samples/detect-api/deidentify-text.ts | 122 - samples/detect-api/get-detect-run.ts | 78 - samples/detect-api/get-run.ts | 90 + samples/detect-api/reidentify-file.ts | 95 + samples/detect-api/reidentify-string.ts | 86 + samples/detect-api/reidentify-text.ts | 88 - samples/vault-api/insert-byot.ts | 4 +- skyvault/README.md | 388 +-- skyvault/src/index.ts | 139 +- skyvault/src/utils/validations/index.ts | 203 -- skyvault/src/vault/client/index.ts | 33 +- skyvault/src/vault/controller/detect/index.ts | 752 +----- skyvault/src/vault/detect/types.ts | 39 + .../deidentify-file/bleep-audio/index.ts | 31 - .../model/options/deidentify-file/index.ts | 147 -- .../model/options/deidentify-text/index.ts | 56 - .../deidentify-text/token-format/index.ts | 45 - .../deidentify-text/transformations/index.ts | 22 - .../model/options/reidentify-text/index.ts | 36 - .../model/request/deidentify-file/index.ts | 19 - .../model/request/deidentify-text/index.ts | 21 - .../model/request/get-detect-run/index.ts | 21 - .../model/request/reidentify-text/index.ts | 21 - .../model/response/deidentify-file/index.ts | 75 - .../model/response/deidentify-text/index.ts | 51 - .../model/response/reidentify-text/index.ts | 16 - skyvault/src/vault/types/index.ts | 49 - skyvault/test/utils/validations.test.js | 772 +----- skyvault/test/vault/client/client.test.js | 35 - skyvault/test/vault/controller/detect.test.js | 2237 +---------------- skyvault/test/vault/model/deprecated.test.js | 92 - 190 files changed, 9454 insertions(+), 6220 deletions(-) create mode 100644 common/src/_generated_/rest/api/errors/InternalServerError.ts create mode 100644 common/src/_generated_/rest/api/resources/authentication/client/requests/GetAuthTokenRequest.ts delete mode 100644 common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/client/Client.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/client/index.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/client/requests/DeidentifyFileRequestV2.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/client/requests/GetRunV2Request.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/client/requests/ReidentifyFileRequestV2.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/client/requests/index.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/index.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataFormat.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataSource.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataFormat.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataSource.ts create mode 100644 common/src/_generated_/rest/api/resources/filesV2/types/index.ts create mode 100644 common/src/_generated_/rest/api/resources/guardrailsV2/client/Client.ts create mode 100644 common/src/_generated_/rest/api/resources/guardrailsV2/client/index.ts create mode 100644 common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/DetectGuardrailsRequestV2.ts create mode 100644 common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/index.ts create mode 100644 common/src/_generated_/rest/api/resources/guardrailsV2/index.ts create mode 100644 common/src/_generated_/rest/api/resources/stringsV2/client/Client.ts create mode 100644 common/src/_generated_/rest/api/resources/stringsV2/client/index.ts create mode 100644 common/src/_generated_/rest/api/resources/stringsV2/client/requests/DeidentifyStringRequestV2.ts create mode 100644 common/src/_generated_/rest/api/resources/stringsV2/client/requests/ReidentifyStringRequestV2.ts create mode 100644 common/src/_generated_/rest/api/resources/stringsV2/client/requests/index.ts create mode 100644 common/src/_generated_/rest/api/resources/stringsV2/index.ts create mode 100644 common/src/_generated_/rest/api/resources/tokenExchange/client/Client.ts create mode 100644 common/src/_generated_/rest/api/resources/tokenExchange/client/index.ts create mode 100644 common/src/_generated_/rest/api/resources/tokenExchange/client/requests/GetStsTokenRequest.ts create mode 100644 common/src/_generated_/rest/api/resources/tokenExchange/client/requests/index.ts create mode 100644 common/src/_generated_/rest/api/resources/tokenExchange/index.ts create mode 100644 common/src/_generated_/rest/api/types/AudioV2.ts create mode 100644 common/src/_generated_/rest/api/types/AudioV2Bleep.ts create mode 100644 common/src/_generated_/rest/api/types/AudioV2OutputTranscription.ts create mode 100644 common/src/_generated_/rest/api/types/DeidentifyFileResponseV2.ts create mode 100644 common/src/_generated_/rest/api/types/DeidentifyStringResponseV2.ts create mode 100644 common/src/_generated_/rest/api/types/DeidentifyStringResponseV2DetectedEntity.ts create mode 100644 common/src/_generated_/rest/api/types/DeidentifyStringResponseV2EntityLocation.ts create mode 100644 common/src/_generated_/rest/api/types/Detect.ts create mode 100644 common/src/_generated_/rest/api/types/DetectConfigV2.ts create mode 100644 common/src/_generated_/rest/api/types/DetectConfigV2VaultType.ts create mode 100644 common/src/_generated_/rest/api/types/DetectCustomTypes.ts create mode 100644 common/src/_generated_/rest/api/types/DetectCustomTypesDeidentificationType.ts create mode 100644 common/src/_generated_/rest/api/types/DetectCustomTypesIncludePrefix.ts create mode 100644 common/src/_generated_/rest/api/types/DetectEntity.ts create mode 100644 common/src/_generated_/rest/api/types/DetectEntityDeidentificationType.ts create mode 100644 common/src/_generated_/rest/api/types/DetectEntityEntityType.ts create mode 100644 common/src/_generated_/rest/api/types/DetectEntityIncludePrefix.ts create mode 100644 common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2.ts create mode 100644 common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2Validation.ts create mode 100644 common/src/_generated_/rest/api/types/DetectObjectEntity.ts create mode 100644 common/src/_generated_/rest/api/types/DetectObjectEntityDeidentificationType.ts create mode 100644 common/src/_generated_/rest/api/types/DetectObjectEntityEntityType.ts create mode 100644 common/src/_generated_/rest/api/types/DetectReturnEntities.ts create mode 100644 common/src/_generated_/rest/api/types/DetectRunsResponseV2.ts create mode 100644 common/src/_generated_/rest/api/types/DetectRunsResponseV2OutputType.ts create mode 100644 common/src/_generated_/rest/api/types/DetectRunsResponseV2Status.ts create mode 100644 common/src/_generated_/rest/api/types/DocumentV2.ts create mode 100644 common/src/_generated_/rest/api/types/DocumentV2Pdf.ts create mode 100644 common/src/_generated_/rest/api/types/DocumentV2PdfProcessingMode.ts create mode 100644 common/src/_generated_/rest/api/types/EntityTransformation.ts rename common/src/_generated_/rest/api/types/{GooglerpcStatus.ts => ErrorResponse.ts} (50%) create mode 100644 common/src/_generated_/rest/api/types/ErrorResponseError.ts create mode 100644 common/src/_generated_/rest/api/types/FileMapping.ts create mode 100644 common/src/_generated_/rest/api/types/FileOutput.ts create mode 100644 common/src/_generated_/rest/api/types/FileOutputProcessedFileExtension.ts create mode 100644 common/src/_generated_/rest/api/types/FileOutputProcessedFileType.ts rename common/src/_generated_/rest/api/types/{V1GetAuthTokenResponse.ts => GetAuthTokenResponse.ts} (81%) create mode 100644 common/src/_generated_/rest/api/types/GetStsTokenResponse.ts create mode 100644 common/src/_generated_/rest/api/types/HttpCode.ts create mode 100644 common/src/_generated_/rest/api/types/ImageV2.ts create mode 100644 common/src/_generated_/rest/api/types/ImageV2MaskingMethod.ts create mode 100644 common/src/_generated_/rest/api/types/Media.ts create mode 100644 common/src/_generated_/rest/api/types/Metrics.ts delete mode 100644 common/src/_generated_/rest/api/types/ProtobufAny.ts create mode 100644 common/src/_generated_/rest/api/types/RedactionLevel.ts create mode 100644 common/src/_generated_/rest/api/types/RedactionLevelEntityName.ts create mode 100644 common/src/_generated_/rest/api/types/RedactionLevelRedactionType.ts create mode 100644 common/src/_generated_/rest/api/types/ReidentifyFileResponseV2.ts create mode 100644 common/src/_generated_/rest/api/types/ReidentifyFileResponseV2OutputType.ts create mode 100644 common/src/_generated_/rest/api/types/ReidentifyFileResponseV2Status.ts create mode 100644 common/src/_generated_/rest/api/types/ReidentifyStringResponseV2.ts create mode 100644 common/src/_generated_/rest/api/types/TransformationShiftDates.ts create mode 100644 common/src/detect/controller/index.ts create mode 100644 common/src/detect/index.ts create mode 100644 common/src/detect/internal/deidentify-file-flow.ts create mode 100644 common/src/detect/internal/errors.ts create mode 100644 common/src/detect/internal/file-output-writer.ts create mode 100644 common/src/detect/internal/request-mapper.ts create mode 100644 common/src/detect/internal/response-mapper.ts create mode 100644 common/src/detect/internal/run-poller.ts create mode 100644 common/src/detect/internal/validations.ts create mode 100644 common/src/detect/model/request/check-guardrails/index.ts create mode 100644 common/src/detect/model/request/deidentify-file/index.ts create mode 100644 common/src/detect/model/request/deidentify-string/index.ts create mode 100644 common/src/detect/model/request/get-run/index.ts create mode 100644 common/src/detect/model/request/reidentify-file/index.ts create mode 100644 common/src/detect/model/request/reidentify-string/index.ts create mode 100644 common/src/detect/model/response/check-guardrails/index.ts create mode 100644 common/src/detect/model/response/deidentify-file/index.ts create mode 100644 common/src/detect/model/response/deidentify-string/index.ts create mode 100644 common/src/detect/model/response/get-run/index.ts create mode 100644 common/src/detect/model/response/reidentify-file/index.ts create mode 100644 common/src/detect/model/response/reidentify-string/index.ts create mode 100644 common/src/detect/types.ts create mode 100644 common/test/detect/controller.test.js create mode 100644 common/test/detect/edge-cases.test.js create mode 100644 common/test/detect/file-output-writer.test.js create mode 100644 common/test/detect/models.test.js create mode 100644 common/test/detect/request-mapper.test.js create mode 100644 common/test/detect/response-mapper.test.js create mode 100644 common/test/detect/run-poller.test.js create mode 100644 common/test/detect/test-types.js create mode 100644 common/test/detect/validations.test.js create mode 100644 flowvault/samples/detect-api/check-guardrails.ts create mode 100644 flowvault/samples/detect-api/deidentify-file-async.ts create mode 100644 flowvault/samples/detect-api/deidentify-file.ts create mode 100644 flowvault/samples/detect-api/deidentify-string.ts create mode 100644 flowvault/samples/detect-api/get-run.ts create mode 100644 flowvault/samples/detect-api/reidentify-file.ts create mode 100644 flowvault/samples/detect-api/reidentify-string.ts create mode 100644 flowvault/src/controller/detect/index.ts create mode 100644 flowvault/src/detect/types.ts create mode 100644 flowvault/test/controller/detect.test.ts create mode 100644 samples/detect-api/check-guardrails.ts create mode 100644 samples/detect-api/deidentify-file-async.ts delete mode 100644 samples/detect-api/deidentify-file-with-filepath-async.ts delete mode 100644 samples/detect-api/deidentify-file-with-filepath.ts create mode 100644 samples/detect-api/deidentify-string.ts delete mode 100644 samples/detect-api/deidentify-text.ts delete mode 100644 samples/detect-api/get-detect-run.ts create mode 100644 samples/detect-api/get-run.ts create mode 100644 samples/detect-api/reidentify-file.ts create mode 100644 samples/detect-api/reidentify-string.ts delete mode 100644 samples/detect-api/reidentify-text.ts create mode 100644 skyvault/src/vault/detect/types.ts delete mode 100644 skyvault/src/vault/model/options/deidentify-file/bleep-audio/index.ts delete mode 100644 skyvault/src/vault/model/options/deidentify-file/index.ts delete mode 100644 skyvault/src/vault/model/options/deidentify-text/index.ts delete mode 100644 skyvault/src/vault/model/options/deidentify-text/token-format/index.ts delete mode 100644 skyvault/src/vault/model/options/deidentify-text/transformations/index.ts delete mode 100644 skyvault/src/vault/model/options/reidentify-text/index.ts delete mode 100644 skyvault/src/vault/model/request/deidentify-file/index.ts delete mode 100644 skyvault/src/vault/model/request/deidentify-text/index.ts delete mode 100644 skyvault/src/vault/model/request/get-detect-run/index.ts delete mode 100644 skyvault/src/vault/model/request/reidentify-text/index.ts delete mode 100644 skyvault/src/vault/model/response/deidentify-file/index.ts delete mode 100644 skyvault/src/vault/model/response/deidentify-text/index.ts delete mode 100644 skyvault/src/vault/model/response/reidentify-text/index.ts diff --git a/README.md b/README.md index 5f765097..55cb491f 100644 --- a/README.md +++ b/README.md @@ -47,10 +47,13 @@ Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow - [Retrieve Existing Tokens: `.tokenize(request)`](#retrieve-existing-tokens-tokenizerequest) - [Construct a `.tokenize()` request](#construct-a-tokenize-request) - [Detect](#detect) - - [De-identify Text: `.deidentifyText(request, options)`](#de-identify-text-deidentifytextrequest-options) - - [Re-identify Text: `.reidentifyText(request, options)`](#re-identify-text-reidentifytextrequest-options) - - [De-identify File: `.deidentifyFile(fileReq, options)`](#de-identify-file-deidentifyfilefilereq-options) - - [Get Run: `.getDetectRun(request)`](#get-run-getdetectrunrequest) + - [De-identify a string: `.deidentifyString(request)`](#de-identify-a-string-deidentifystringrequest) + - [Detect configuration](#detect-configuration) + - [De-identify a file: `.deidentifyFile(request)`](#de-identify-a-file-deidentifyfilerequest) + - [Re-identify a string: `.reidentifyString(request)`](#re-identify-a-string-reidentifystringrequest) + - [Re-identify a file: `.reidentifyFile(request)`](#re-identify-a-file-reidentifyfilerequest) + - [Get run: `.getRun(request)`](#get-run-getrunrequest) + - [Check guardrails: `.checkGuardrails(request)`](#check-guardrails-checkguardrailsrequest) - [Connections](#connections) - [Configure a connection](#configure-a-connection) - [Invoke a connection](#invoke-a-connection) @@ -627,213 +630,288 @@ Tokenization Result: { tokens: ['5391-4629-3722-7102', '0f6b8a2c-90ab-4cde-9def- ## Detect -De-identify and reidentify sensitive data in text and files using Skyflow Detect, which supports advanced privacy-preserving workflows. +Skyflow Detect finds and de-identifies sensitive data in free text and files, then re-identifies it on demand. The SDK exposes the Detect V2 API through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault and uses the vault's credentials and log level. -### De-identify Text: `.deidentifyText(request, options)` +Detect configurations are created in Studio or through the management API. The SDK does not create, update or delete them. -De-identify or anonymize text using the `deidentifyText` method. +| Method | Request | Response | Description | +|--------|---------|----------|-------------| +| `deidentifyString(request)` | [`DeidentifyStringRequest`](docs/api_reference.md#deidentifystringrequest) | [`DeidentifyStringResponse`](docs/api_reference.md#deidentifystringresponse) | Detect and replace sensitive data in a string | +| `deidentifyFile(request)` | [`DeidentifyFileRequest`](docs/api_reference.md#deidentifyfilerequest) | [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) | Submit a file for de-identification, optionally polling the run to completion | +| `reidentifyString(request)` | [`ReidentifyStringRequest`](docs/api_reference.md#reidentifystringrequest) | [`ReidentifyStringResponse`](docs/api_reference.md#reidentifystringresponse) | Replace tokens in a string with their original or redacted values | +| `reidentifyFile(request)` | [`ReidentifyFileRequest`](docs/api_reference.md#reidentifyfilerequest) | [`ReidentifyFileResponse`](docs/api_reference.md#reidentifyfileresponse) | Replace tokens in a file with their original or redacted values | +| `getRun(request)` | [`GetRunRequest`](docs/api_reference.md#getrunrequest) | [`GetRunResponse`](docs/api_reference.md#getrunresponse) | Retrieve the status and outputs of a Detect file run | +| `checkGuardrails(request)` | [`CheckGuardrailsRequest`](docs/api_reference.md#checkguardrailsrequest) | [`CheckGuardrailsResponse`](docs/api_reference.md#checkguardrailsresponse) | Screen text for toxicity and denied topics | -Create a de-identify text request with the [`DeidentifyTextRequest`](docs/api_reference.md#deidentifytextrequest) class, which includes the text to be deidentified. Provide optional parameters using the [`DeidentifyTextOptions`](docs/api_reference.md#deidentifytextoptions) class. +**Before you start** + +- **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy that grants the Detokenization action on the destination table and columns, unless it is a Vault Owner. +- **Configuration.** `configurationId` is the id of a Detect configuration created through the management API. The alternative is an inline `DetectConfiguration` passed as `configuration` (see [Detect configuration](#detect-configuration)). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings. Sending both is rejected by the API with HTTP 400. +- **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the `table.column` named by `destination`. That table and column must already exist in the vault. +- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](docs/api_reference.md#detect-api). + +### De-identify a string: `.deidentifyString(request)` + +`DeidentifyStringRequest` takes the text plus, optionally, `configurationId` (a stored Detect configuration) or `configuration` (an inline `DetectConfiguration`). The SDK fills the vault id into the configuration from the controller. ```typescript import { - DeidentifyTextRequest, - DeidentifyTextOptions, + DeidentifyStringRequest, + DeidentifyStringResponse, + DetectConfiguration, + EntityType, + DeidentificationType, + ReturnEntitiesType, SkyflowError, - TokenFormat, - TokenType, - Transformations, - DetectEntities, -} from "skyflow-node"; +} from 'skyflow-node'; -// Prepare the text to be deidentified -const deidentifyTextRequest = new DeidentifyTextRequest( - "", +// Stored configuration +const stored = new DeidentifyStringRequest( + 'My name is John Doe, and my email is johndoe@acme.com.', + { configurationId: '' }, ); - -// Configure DeidentifyTextOptions -const options = new DeidentifyTextOptions(); -options.setEntities([DetectEntities.ACCOUNT_NUMBER, DetectEntities.SSN]); // Entities to de-identify -options.setAllowRegexList([""]); // Allowlist regex patterns -options.setRestrictRegexList([""]); // Restrict regex patterns - -const tokenFormat = new TokenFormat(); -tokenFormat.setDefault(TokenType.VAULT_TOKEN); // default format -options.setTokenFormat(tokenFormat); - -const transformations = new Transformations(); -transformations.setShiftDays({ - max: 30, // maximum days to shift - min: 10, // minimum days to shift - entities: [DetectEntities.ACCOUNT_NUMBER, DetectEntities.SSN], // entity types to apply the shift -}); -options.setTransformations(transformations); - -// Call deidentifyText -const response = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyText(deidentifyTextRequest, options); - -console.log("De-identify Text Response:", response); +const response: DeidentifyStringResponse = await skyflowClient.detect().deidentifyString(stored); +console.log(response.processedText); +// My name is [NAME_1], and my email is [EMAIL_ADDRESS_1]. + +// Inline configuration +const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '.', // table.column that stores the value + }, + { entityType: EntityType.ALL, deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER }, + ], + skip: ['Skyflow'], + returnEntities: ReturnEntitiesType.EXCLUDE_SENSITIVE_DATA, + }, +}; +const inline = new DeidentifyStringRequest( + 'My name is John Doe, and my email is johndoe@acme.com.', + { configuration }, +); +const inlineResponse = await skyflowClient.detect().deidentifyString(inline); +for (const entity of inlineResponse.entities) { + console.log(entity.entityType, entity.token, entity.location?.startIndex, entity.location?.endIndex); +} +console.log(inlineResponse.metrics?.wordCount); ``` -Returns a [`DeidentifyTextResponse`](docs/api_reference.md#deidentifytextresponse) (`processedText`, `entities`, `wordCount`, `charCount`, `errors`): - -```text -De-identify Text Response: { processedText: 'Hello, [PERSON_1]! My SSN is [SSN_1].', entities: [{ token: '[PERSON_1]', value: 'John', entity: 'NAME', scores: { NAME: 0.98 } }], wordCount: 8, charCount: 37, errors: null } -``` +`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. > [!TIP] -> See the full example in the samples directory: [deidentify-text.ts](samples/detect-api/deidentify-text.ts) +> See the full example in the samples directory: [deidentify-string.ts](samples/detect-api/deidentify-string.ts) -### Re-identify Text: `.reidentifyText(request, options)` +### Detect configuration -Re-identify text using the `reidentifyText` method. Create a reidentify text request with the [`ReidentifyTextRequest`](docs/api_reference.md#reidentifytextrequest) class, which includes the redacted or de-identified text to be re-identified. Provide optional parameters using the [`ReidentifyTextOptions`](docs/api_reference.md#reidentifytextoptions) class to control how specific entities are returned (as redacted, masked, or plain text). +`DetectConfiguration` is the inline form of a stored configuration and accepts the same options. `detect` applies to string and file inputs; `fileMapping` and `media` apply to file inputs only. All of these are plain TypeScript interfaces, so write them as object literals. + +| Type | Fields | Notes | +|------|--------|-------| +| `Detect` | `entities`, `objectEntities`, `restrict`, `skip`, `customTypes`, `returnEntities` | `restrict` and `skip` are regexes that are always, respectively never, treated as sensitive. `returnEntities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. | +| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` is used only with `VAULT_TOKEN`. | +| `Transformation` / `ShiftDates` | `shiftDates`; `minDays`, `maxDays` | Shifts detected dates by a random number of days within the range. | +| `CustomType` | `label`, `match` | A regex-defined entity type. `label` is reported as the entity type of its matches. | +| `ObjectEntity` | `entityType`, `deidentificationType` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. | +| `FileMapping` | `source`, `destination`, `outputFileNameSuffix`, `entitiesDestination`, `objectEntitiesDestination` | Where the processed file and the entity files are written. File inputs only. | +| `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. | +| `Audio` / `Bleep` | `outputProcessedAudio`, `outputTranscription`, `bleep`; `startPadding`, `stopPadding`, `frequency`, `gain` | `outputTranscription` is an `OutputTranscriptionType`. `Bleep` is the tone laid over detected speech; paddings are in seconds, frequency in Hz. | +| `Document` / `Pdf` | `pdf`; `processingMode`, `density`, `maxResolution` | `processingMode` is a `PdfProcessingMode` (`NONE`, `OCR` or `TEXT_LAYER`). `density` is the rendering DPI. | +| `Image` | `outputProcessedImage`, `outputOcrText`, `maskingMethod` | `maskingMethod` is a `MaskingMethod` (`NONE`, `BLUR` or `BLACKBOX`). | ```typescript import { - ReidentifyTextRequest, - ReidentifyTextOptions, - SkyflowError, - DetectEntities, - ReidentifyTextResponse, -} from "skyflow-node"; + DetectConfiguration, + EntityType, + DeidentificationType, + ObjectEntityType, + ObjectDeidentificationType, + MaskingMethod, + PdfProcessingMode, + OutputTranscriptionType, +} from 'skyflow-node'; + +const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.DOB, + deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, + transformation: { shiftDates: { minDays: 10, maxDays: 30 } }, + }, + { entityType: EntityType.ALL }, + ], + objectEntities: [{ entityType: ObjectEntityType.FACE, deidentificationType: ObjectDeidentificationType.REDACT }], + customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], + restrict: ['[0-9]{3}-[0-9]{2}-[0-9]{4}'], + }, + media: { + image: { maskingMethod: MaskingMethod.BLUR, outputOcrText: true }, + document: { pdf: { processingMode: PdfProcessingMode.OCR } }, + audio: { + outputTranscription: OutputTranscriptionType.TRANSCRIPTION, + bleep: { startPadding: 0.5, stopPadding: 0.5 }, + }, + }, +}; +``` -// Prepare the redacted text to be re-identified -const request = new ReidentifyTextRequest(""); +### De-identify a file: `.deidentifyFile(request)` -// Configure ReidentifyTextOptions -const options = new ReidentifyTextOptions(); -options.setRedactedEntities([DetectEntities.SSN]); // Entities to keep redacted -options.setMaskedEntities([DetectEntities.CREDIT_CARD_NUMBER]); // Entities to mask -options.setPlainTextEntities([DetectEntities.NAME]); // Entities to return as plain text +File de-identification is asynchronous on the API: the submit call returns a run id and the result is retrieved from the run. `deidentifyFile` exposes both modes. -// Call reidentifyText -const response: ReidentifyTextResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .reidentifyText(request, options); +- **Submit only** (default): one call, returns `runId`. `status` is `null` and nothing is written to disk. Retrieve the result later with [`getRun`](#get-run-getrunrequest). +- **Submit and poll**: set `pollOptions`. The SDK polls the run with exponential backoff (1, 2, 4, 8, 16, then 32 seconds between lookups) until it is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `waitTime` (default 60 s, max 300) or `maxAttempts` (default 23, max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED` so you can resume with `getRun`. -console.log("Reidentify Text Response:", response); -``` +The file is supplied in one of three ways: a `dataSource` plus `value` (`BASE64` content, a `SKYFLOW_ID`, or a `PRESIGNED_URL`), a `file` (a web `File`, Node.js 20+), or a `filePath`. A local file is read, base64-encoded, and its `dataFormat` inferred from the extension. Supply exactly one of `value`, `file` or `filePath`. `configurationId` and `configuration` follow the same rules as for strings. -Returns a [`ReidentifyTextResponse`](docs/api_reference.md#reidentifytextresponse) (`processedText`): +`dataFormat` accepts every `FileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. -```text -Reidentify Text Response: { processedText: 'Hello, John! My SSN is 123-45-6789.' } +`outputDirectory` is meaningful only when polling. Setting it without `pollOptions` fails client-side validation before any request is sent. + +```typescript +import { DeidentifyFileRequest, DeidentifyFileResponse, DetectRunStatus } from 'skyflow-node'; + +// Submit only +const submitted: DeidentifyFileResponse = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ + filePath: '/data/intake/patient-notes.pdf', // sent as BASE64, format inferred as PDF + configurationId: '', + }), +); +const runId = submitted.runId; + +// Submit and poll +const response = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ + filePath: '/data/intake/patient-notes.pdf', + configurationId: '', + pollOptions: { waitTime: 120 }, + outputDirectory: '/data/processed', // optional + }), +); + +switch (response.status) { + case DetectRunStatus.SUCCESS: + // For local-file requests the outputs are already on disk: + // /data/processed/processed-patient-notes.pdf the redacted file + // /data/processed/processed-entities-patient-notes.json the detected entities + // response.output and response.metrics carry the same data. + break; + case DetectRunStatus.FAILED: + console.error(response.message); + break; + default: // IN_PROGRESS or QUEUED: budget ran out, resume with getRun(runId) + break; +} ``` +Outputs are written only when polling reached `SUCCESS`, the request was built from a local file, and the run delivered them as base64. A request built from a `SKYFLOW_ID` or `PRESIGNED_URL` has no local name to derive from, and presigned-URL outputs are not fetched, so in those cases read `output` and write the files yourself. Files are named `processed-.`, with the entities output as `processed-entities-.json`; `outputDirectory` is created if missing and defaults to the working directory. Writing outputs needs the Node.js file system and is not supported in Cloudflare Workers. + > [!TIP] -> See the full example in the samples directory: [reidentify-text.ts](samples/detect-api/reidentify-text.ts) +> See the full examples in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](samples/detect-api/deidentify-file-async.ts) -### De-identify File: `.deidentifyFile(fileReq, options)` +### Re-identify a string: `.reidentifyString(request)` -De-identify files using the `deidentifyFile` method. Create a de-identify file request with the [`DeidentifyFileRequest`](docs/api_reference.md#deidentifyfilerequest) class, which includes the file to be deidentified (such as images, PDFs, audio, documents, spreadsheets, or presentations). Provide optional parameters using the [`DeidentifyFileOptions`](docs/api_reference.md#deidentifyfileoptions) class to control how entities are detected and deidentified, as well as how the output is generated for different file types. +Re-identification looks tokens up in the vault, so it applies to values that were de-identified with `VAULT_TOKEN`. Placeholders produced by `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not stored anywhere and cannot be re-identified. -> [!NOTE] -> File de-identification requires Node.js v20.x or above. +`ReidentifyStringRequest` takes the tokenized text and an optional list of `RedactionLevel` rules. For skyvault vaults each rule names an `entityName` and the `redactionType` to render it with. Without rules every token is replaced by its original value. ```typescript -import { - DeidentifyFileRequest, - DeidentifyFileOptions, - DeidentifyFileResponse - SkyflowError, - DetectEntities, - MaskingMethod, - DetectOutputTranscription, - Bleep -} from 'skyflow-node'; +import { ReidentifyStringRequest, ReidentifyStringResponse, EntityType, RedactionType } from 'skyflow-node'; -// Prepare the file to be deidentified -const filePath: string = ''; -const buffer = fs.readFileSync(filePath); -const file = new File([buffer], filePath); +const response: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest('My name is [NAME_1].', { + redactionLevel: [{ entityName: EntityType.NAME, redactionType: RedactionType.MASKED }], + }), +); +console.log(response.processedText); +// My name is J**n D*e. +``` -// Construct the file input by providing either a file object or file path, but not both -const fileInput: FileInput = { file: file } // OR const fileInput: FileInput = { filePath: filePath } -const fileReq = new DeidentifyFileRequest(fileInput); +`RedactionLevel` is the skyvault SDK's own type and has only `entityName` and `redactionType`; the `tokenGroupName` and `redactionPattern` rules of the flowvault SDK do not exist here. Both fields are optional on the API, so the SDK passes the list through as given and the server rejects invalid combinations. -// Configure DeidentifyFileOptions -const options = new DeidentifyFileOptions(); -options.setEntities([DetectEntities.SSN, DetectEntities.ACCOUNT_NUMBER]); +> [!TIP] +> See the full example in the samples directory: [reidentify-string.ts](samples/detect-api/reidentify-string.ts) -const tokenFormat = new TokenFormat(); // Token format for deidentified entities -tokenFormat.setDefault(TokenType.ENTITY_ONLY); -options.setTokenFormat(tokenFormat); +### Re-identify a file: `.reidentifyFile(request)` +`reidentifyFile` is synchronous: the re-identified file comes back in the response, not through a run id. `ReidentifyFileRequest` takes a `dataSource` (`ReidentifySourceType.BASE64` or `PRESIGNED_URL`), the matching `value`, an optional `dataFormat` and the same optional `RedactionLevel` rules as `reidentifyString`. `dataFormat` is limited to the formats that support `VAULT_TOKEN`: `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. -options.setOutputDirectory(''); // Output directory for saving the deidentified file. Not supported in Cloudflare workers. -options.setWaitTime(64); // Wait time for polling (max 64 seconds; returns runId + status if exceeded) +```typescript +import * as fs from 'fs'; +import { ReidentifyFileRequest, ReidentifySourceType, FileDataFormat, DetectRunStatus } from 'skyflow-node'; +const content = fs.readFileSync('tokenized.txt').toString('base64'); -tifyFile -const response: DeidentifyFileResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyFile(fileReq, options); +const response = await skyflowClient.detect().reidentifyFile( + new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: FileDataFormat.TXT }), +); -console.log('De-identify File Response:', response); +if (response.status === DetectRunStatus.SUCCESS) { + for (const output of response.output) { + // output.processedFile is base64 or a presigned URL, per response.outputType + const processed = Buffer.from(output.processedFile ?? '', 'base64'); + } +} ``` -Returns a [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) (`fileBase64`, `file`, `runId`, `status`, `errors`). If processing exceeds `setWaitTime`, only `runId` and `status` are populated — poll with `getDetectRun`: - -```text -De-identify File Response: { runId: 'run-abc123', status: 'IN_PROGRESS', fileBase64: undefined, errors: null } -``` +`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. -**Supported file types:** +> [!TIP] +> See the full example in the samples directory: [reidentify-file.ts](samples/detect-api/reidentify-file.ts) -- Documents: `doc`, `docx`, `pdf` -- PDFs: `pdf` -- Images: `bmp`, `jpeg`, `jpg`, `png`, `tif`, `tiff` -- Structured text: `json`, `xml` -- Spreadsheets: `csv`, `xls`, `xlsx` -- Presentations: `ppt`, `pptx` -- Audio: `mp3`, `wav` +### Get run: `.getRun(request)` -**Notes:** -- Transformations can't be applied to Documents, Images, or PDFs. -- `setWaitTime` must be ≤ 64 seconds; otherwise, an error is thrown. -- If the API takes more than `waitTime` seconds, the response contains only `runId` and `status`. Use `.getDetectRun()` to poll for the result. +File de-identification is asynchronous: the submit call returns a run id, and `getRun` retrieves the run's status and, once it has succeeded, its outputs. `output` is empty and `outputType` and `metrics` are `null` until the run completes. -For all `DeidentifyFileOptions` setters and the `Bleep` class, see [DeidentifyFileOptions](docs/api_reference.md#deidentifyfileoptions) in the API reference. +```typescript +import { GetRunRequest, GetRunResponse, DetectRunStatus } from 'skyflow-node'; + +const run: GetRunResponse = await skyflowClient.detect().getRun(new GetRunRequest('')); + +switch (run.status) { + case DetectRunStatus.SUCCESS: + for (const output of run.output) { + // output.processedFile is base64 or a presigned URL, per run.outputType + // output.processedFileType is e.g. REDACTED_FILE or ENTITIES + // output.processedFileExtension is a FileDataFormat such as PDF + } + break; + case DetectRunStatus.FAILED: + console.error(run.message); + break; + default: // QUEUED, IN_PROGRESS, UNKNOWN: poll again later + break; +} +``` > [!TIP] -> See the full example in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) +> See the full example in the samples directory: [get-run.ts](samples/detect-api/get-run.ts) -### Get Run: `.getDetectRun(request)` +### Check guardrails: `.checkGuardrails(request)` -Retrieve the results of a previously started file de-identification operation (or 'run') using the `getDetectRun(...)` method. Initialize the [`GetDetectRunRequest`](docs/api_reference.md#getdetectrunrequest) class with the `runId` returned from a prior `.deidentifyFile(fileReq, options)` call. Fetch the final results of the file de-identification operation once they are available. +`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. ```typescript -import { - GetDetectRunRequest, - DeidentifyFileResponse, - DeidentifyFileResponse - SkyflowError -} from 'skyflow-node'; +import { CheckGuardrailsRequest, GuardrailsValidation } from 'skyflow-node'; -// Prepare the GetDetectRunRequest with the runId from a previous deidentifyFile call -const request = new GetDetectRunRequest({ - runId: '', // Replace with the runId you received earlier -}); - -// Step 2: Call getDetectRun -const response: DeidentifyFileResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .getDetectRun(request); +const response = await skyflowClient.detect().checkGuardrails( + new CheckGuardrailsRequest(userPrompt, { checkToxicity: true, denyTopics: ['politics', 'medical advice'] }), +); -// Step 3: Handle the response -console.log('Get Detect Run Response:', response); +if (response.validation === GuardrailsValidation.FAILED) { + throw new Error(`blocked: toxic=${response.toxic} deniedTopic=${response.deniedTopic}`); +} ``` -Returns a [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) with the current `status` (and the processed file once complete): - -```text -Get Detect Run Response: { status: 'SUCCESS', runId: 'run-abc123', fileBase64: '', errors: null } -``` +All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling). Client-side validation fails before any network call. > [!TIP] -> See the full example in the samples directory: [get-detect-run.ts](samples/detect-api/get-detect-run.ts) +> See the full example in the samples directory: [check-guardrails.ts](samples/detect-api/check-guardrails.ts) ## Connections @@ -1307,7 +1385,11 @@ import { // Request item shapes DetokenizeData, // { token: string; redactionType?: RedactionType } TokenizeRequestType, // { value: string; columnGroup: string } - FileInput, // { file: File } | { filePath: string } + + // Detect configuration and rule shapes + DetectConfiguration, // inline Detect configuration: { detect?, fileMapping?, media? } + RedactionLevel, // { entityName?, redactionType? } (skyvault); the flowvault SDK has { tokenGroupName?, redactionPattern? } + PollOptions, // { waitTime?: number; maxAttempts?: number } // Response record shapes InsertResponseType, // { skyflowId: string; [field: string]: unknown } diff --git a/common/src/_generated_/rest/Client.ts b/common/src/_generated_/rest/Client.ts index b278e5b7..00d29978 100644 --- a/common/src/_generated_/rest/Client.ts +++ b/common/src/_generated_/rest/Client.ts @@ -8,13 +8,17 @@ import * as Skyflow from "./api/index"; import urlJoin from "url-join"; import * as errors from "./errors/index"; import { Authentication } from "./api/resources/authentication/client/Client"; +import { TokenExchange } from "./api/resources/tokenExchange/client/Client"; +import { FilesV2 } from "./api/resources/filesV2/client/Client"; +import { StringsV2 } from "./api/resources/stringsV2/client/Client"; +import { GuardrailsV2 } from "./api/resources/guardrailsV2/client/Client"; export declare namespace SkyflowClient { export interface Options { environment?: core.Supplier; /** Specify a custom URL to connect the client to. */ baseUrl?: core.Supplier; - token: core.Supplier; + token?: core.Supplier; fetcher?: core.FetchFunction; } @@ -32,13 +36,33 @@ export declare namespace SkyflowClient { export class SkyflowClient { protected _authentication: Authentication | undefined; + protected _tokenExchange: TokenExchange | undefined; + protected _filesV2: FilesV2 | undefined; + protected _stringsV2: StringsV2 | undefined; + protected _guardrailsV2: GuardrailsV2 | undefined; - constructor(protected readonly _options: SkyflowClient.Options) {} + constructor(protected readonly _options: SkyflowClient.Options = {}) {} public get authentication(): Authentication { return (this._authentication ??= new Authentication(this._options)); } + public get tokenExchange(): TokenExchange { + return (this._tokenExchange ??= new TokenExchange(this._options)); + } + + public get filesV2(): FilesV2 { + return (this._filesV2 ??= new FilesV2(this._options)); + } + + public get stringsV2(): StringsV2 { + return (this._stringsV2 ??= new StringsV2(this._options)); + } + + public get guardrailsV2(): GuardrailsV2 { + return (this._guardrailsV2 ??= new GuardrailsV2(this._options)); + } + /** * @param {string | undefined} vaultId * @param {Skyflow.PatchV2VaultsIdRequest} request @@ -113,7 +137,12 @@ export class SkyflowClient { } } - protected async _getAuthorizationHeader(): Promise { - return `Bearer ${await core.Supplier.get(this._options.token)}`; + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; } } diff --git a/common/src/_generated_/rest/api/errors/BadRequestError.ts b/common/src/_generated_/rest/api/errors/BadRequestError.ts index e19131a1..1f9b3de6 100644 --- a/common/src/_generated_/rest/api/errors/BadRequestError.ts +++ b/common/src/_generated_/rest/api/errors/BadRequestError.ts @@ -3,10 +3,11 @@ */ import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; import * as core from "../../core"; export class BadRequestError extends errors.SkyflowError { - constructor(body: Record, rawResponse?: core.RawResponse) { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { super({ message: "BadRequestError", statusCode: 400, diff --git a/common/src/_generated_/rest/api/errors/InternalServerError.ts b/common/src/_generated_/rest/api/errors/InternalServerError.ts new file mode 100644 index 00000000..4141147e --- /dev/null +++ b/common/src/_generated_/rest/api/errors/InternalServerError.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; +import * as core from "../../core"; + +export class InternalServerError extends errors.SkyflowError { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { + super({ + message: "InternalServerError", + statusCode: 500, + body: body, + rawResponse: rawResponse, + }); + Object.setPrototypeOf(this, InternalServerError.prototype); + } +} diff --git a/common/src/_generated_/rest/api/errors/NotFoundError.ts b/common/src/_generated_/rest/api/errors/NotFoundError.ts index ff59e84a..f5226d35 100644 --- a/common/src/_generated_/rest/api/errors/NotFoundError.ts +++ b/common/src/_generated_/rest/api/errors/NotFoundError.ts @@ -3,10 +3,11 @@ */ import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; import * as core from "../../core"; export class NotFoundError extends errors.SkyflowError { - constructor(body: Record, rawResponse?: core.RawResponse) { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { super({ message: "NotFoundError", statusCode: 404, diff --git a/common/src/_generated_/rest/api/errors/UnauthorizedError.ts b/common/src/_generated_/rest/api/errors/UnauthorizedError.ts index 73c2b114..9d763249 100644 --- a/common/src/_generated_/rest/api/errors/UnauthorizedError.ts +++ b/common/src/_generated_/rest/api/errors/UnauthorizedError.ts @@ -3,10 +3,11 @@ */ import * as errors from "../../errors/index"; +import * as Skyflow from "../index"; import * as core from "../../core"; export class UnauthorizedError extends errors.SkyflowError { - constructor(body: Record, rawResponse?: core.RawResponse) { + constructor(body: Skyflow.ErrorResponse, rawResponse?: core.RawResponse) { super({ message: "UnauthorizedError", statusCode: 401, diff --git a/common/src/_generated_/rest/api/errors/index.ts b/common/src/_generated_/rest/api/errors/index.ts index 91d5e383..8d78fac7 100644 --- a/common/src/_generated_/rest/api/errors/index.ts +++ b/common/src/_generated_/rest/api/errors/index.ts @@ -1,3 +1,4 @@ export * from "./BadRequestError"; export * from "./UnauthorizedError"; export * from "./NotFoundError"; +export * from "./InternalServerError"; diff --git a/common/src/_generated_/rest/api/index.ts b/common/src/_generated_/rest/api/index.ts index dba65abb..d5c609c4 100644 --- a/common/src/_generated_/rest/api/index.ts +++ b/common/src/_generated_/rest/api/index.ts @@ -1,4 +1,4 @@ +export * from "./resources"; export * from "./types"; export * from "./errors"; -export * from "./resources"; export * from "./client"; diff --git a/common/src/_generated_/rest/api/resources/authentication/client/Client.ts b/common/src/_generated_/rest/api/resources/authentication/client/Client.ts index 7c3b0039..63db1d87 100644 --- a/common/src/_generated_/rest/api/resources/authentication/client/Client.ts +++ b/common/src/_generated_/rest/api/resources/authentication/client/Client.ts @@ -13,7 +13,7 @@ export declare namespace Authentication { environment?: core.Supplier; /** Specify a custom URL to connect the client to. */ baseUrl?: core.Supplier; - token: core.Supplier; + token?: core.Supplier; fetcher?: core.FetchFunction; } @@ -30,35 +30,36 @@ export declare namespace Authentication { } export class Authentication { - constructor(protected readonly _options: Authentication.Options) {} + constructor(protected readonly _options: Authentication.Options = {}) {} /** - *

Generates a Bearer Token to authenticate with Skyflow. This method doesn't require the Authorization header.

Note: For recommended ways to authenticate, see API authentication.

+ * Generates a Bearer Token to authenticate with Skyflow. * - * @param {Skyflow.V1GetAuthTokenRequest} request + * @param {Skyflow.GetAuthTokenRequest} request * @param {Authentication.RequestOptions} requestOptions - Request-specific configuration. * * @throws {@link Skyflow.BadRequestError} * @throws {@link Skyflow.UnauthorizedError} * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.InternalServerError} * * @example - * await client.authentication.authenticationServiceGetAuthToken({ - * grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer", - * assertion: "eyLhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaXNzIjoiY29tcGFueSIsImV4cCI6MTYxNTE5MzgwNywiaWF0IjoxNjE1MTY1MDQwLCJhdWQiOiKzb21lYXVkaWVuY2UifQ.4pcPyMDQ9o1PSyXnrXCjTwXyr4BSezdI1AVTmud2fU3" + * await client.authentication.getBearerToken({ + * grant_type: "grant_type", + * assertion: "assertion" * }) */ - public authenticationServiceGetAuthToken( - request: Skyflow.V1GetAuthTokenRequest, + public getBearerToken( + request: Skyflow.GetAuthTokenRequest, requestOptions?: Authentication.RequestOptions, - ): core.HttpResponsePromise { - return core.HttpResponsePromise.fromPromise(this.__authenticationServiceGetAuthToken(request, requestOptions)); + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__getBearerToken(request, requestOptions)); } - private async __authenticationServiceGetAuthToken( - request: Skyflow.V1GetAuthTokenRequest, + private async __getBearerToken( + request: Skyflow.GetAuthTokenRequest, requestOptions?: Authentication.RequestOptions, - ): Promise> { + ): Promise> { const _response = await (this._options.fetcher ?? core.fetcher)({ url: urlJoin( (await core.Supplier.get(this._options.baseUrl)) ?? @@ -85,24 +86,29 @@ export class Authentication { abortSignal: requestOptions?.abortSignal, }); if (_response.ok) { - return { data: _response.body as Skyflow.V1GetAuthTokenResponse, rawResponse: _response.rawResponse }; + return { data: _response.body as Skyflow.GetAuthTokenResponse, rawResponse: _response.rawResponse }; } if (_response.error.reason === "status-code") { switch (_response.error.statusCode) { case 400: throw new Skyflow.BadRequestError( - _response.error.body as Record, + _response.error.body as Skyflow.ErrorResponse, _response.rawResponse, ); case 401: throw new Skyflow.UnauthorizedError( - _response.error.body as Record, + _response.error.body as Skyflow.ErrorResponse, _response.rawResponse, ); case 404: throw new Skyflow.NotFoundError( - _response.error.body as Record, + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, _response.rawResponse, ); default: @@ -131,7 +137,12 @@ export class Authentication { } } - protected async _getAuthorizationHeader(): Promise { - return `Bearer ${await core.Supplier.get(this._options.token)}`; + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; } } diff --git a/common/src/_generated_/rest/api/resources/authentication/client/requests/GetAuthTokenRequest.ts b/common/src/_generated_/rest/api/resources/authentication/client/requests/GetAuthTokenRequest.ts new file mode 100644 index 00000000..42f11274 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/authentication/client/requests/GetAuthTokenRequest.ts @@ -0,0 +1,25 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * { + * grant_type: "grant_type", + * assertion: "assertion" + * } + */ +export interface GetAuthTokenRequest { + /** Grant type of the request. Set this to `urn:ietf:params:oauth:grant-type:jwt-bearer`. */ + grant_type: string; + /** User-signed JWT token that contains the following fields: iss (Issuer), key (Unique identifier), aud (Recipient), exp (Expiry), sub (Subject), ctx (Optional context for context-aware authorization). */ + assertion: string; + /** Subject token. */ + subject_token?: string; + /** Subject token type. */ + subject_token_type?: string; + /** Token use type. Either `delegation` or `impersonation`. */ + requested_token_use?: string; + /** Subset of available roles to associate with the requested token. Uses the format "role: role:". */ + scope?: string; +} diff --git a/common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts b/common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts deleted file mode 100644 index a1ef4fba..00000000 --- a/common/src/_generated_/rest/api/resources/authentication/client/requests/V1GetAuthTokenRequest.ts +++ /dev/null @@ -1,25 +0,0 @@ -/** - * This file was auto-generated by Fern from our API Definition. - */ - -/** - * @example - * { - * grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer", - * assertion: "eyLhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaXNzIjoiY29tcGFueSIsImV4cCI6MTYxNTE5MzgwNywiaWF0IjoxNjE1MTY1MDQwLCJhdWQiOiKzb21lYXVkaWVuY2UifQ.4pcPyMDQ9o1PSyXnrXCjTwXyr4BSezdI1AVTmud2fU3" - * } - */ -export interface V1GetAuthTokenRequest { - /** Grant type of the request. Set this to `urn:ietf:params:oauth:grant-type:jwt-bearer`. */ - grant_type: string; - /** User-signed JWT token that contains the following fields:
  • iss: Issuer of the JWT.
  • key: Unique identifier for the key.
  • aud: Recipient the JWT is intended for.
  • exp: Time the JWT expires.
  • sub: Subject of the JWT.
  • ctx: (Optional) Value for Context-aware authorization.
*/ - assertion: string; - /** Subject token. */ - subject_token?: string; - /** Subject token type. */ - subject_token_type?: string; - /** Token use type. Either `delegation` or `impersonation`. */ - requested_token_use?: string; - /** Subset of available roles to associate with the requested token. Uses the format "role:\ role:\". */ - scope?: string; -} diff --git a/common/src/_generated_/rest/api/resources/authentication/client/requests/index.ts b/common/src/_generated_/rest/api/resources/authentication/client/requests/index.ts index 635fca22..03b9adac 100644 --- a/common/src/_generated_/rest/api/resources/authentication/client/requests/index.ts +++ b/common/src/_generated_/rest/api/resources/authentication/client/requests/index.ts @@ -1 +1 @@ -export { type V1GetAuthTokenRequest } from "./V1GetAuthTokenRequest"; +export { type GetAuthTokenRequest } from "./GetAuthTokenRequest"; diff --git a/common/src/_generated_/rest/api/resources/filesV2/client/Client.ts b/common/src/_generated_/rest/api/resources/filesV2/client/Client.ts new file mode 100644 index 00000000..b9202768 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/client/Client.ts @@ -0,0 +1,412 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace FilesV2 { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class FilesV2 { + constructor(protected readonly _options: FilesV2.Options = {}) {} + + /** + * De-identifies sensitive data from a file.

Note: + * - Layout may vary with token-based redaction for MS Office files (PPT, DOCX, and XLS) and PDFs when token-based PDF processing is selected. + * - V2 API is in beta. + * + * @param {Skyflow.DeidentifyFileRequestV2} request + * @param {FilesV2.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.filesV2.deidentifyFileV2({ + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "pdf", + * configuration: { + * vaultId: "$VAULT_ID", + * detect: { + * entities: [{ + * entityType: "NAME", + * deidentificationType: "VAULT_TOKEN", + * destination: "persons.name" + * }, { + * entityType: "DOB", + * deidentificationType: "ENTITY_UNIQUE_COUNTER", + * transformation: { + * shiftDates: { + * minDays: 10, + * maxDays: 30 + * } + * } + * }, { + * entityType: "ALL", + * deidentificationType: "ENTITY_UNIQUE_COUNTER" + * }], + * restrict: ["restrict"], + * skip: ["skip"], + * customTypes: [{ + * label: "EMPLOYEE_ID", + * match: ["EMP-[0-9]{6}"] + * }], + * returnEntities: "EXCLUDE_SENSITIVE_DATA" + * }, + * media: { + * document: { + * pdf: { + * processingMode: "TEXT_LAYER", + * density: 200, + * maxResolution: 3000 + * } + * } + * } + * } + * }) + * + * @example + * await client.filesV2.deidentifyFileV2({ + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "pdf", + * configurationId: "$CONFIGURATION_ID" + * }) + */ + public deidentifyFileV2( + request: Skyflow.DeidentifyFileRequestV2, + requestOptions?: FilesV2.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__deidentifyFileV2(request, requestOptions)); + } + + private async __deidentifyFileV2( + request: Skyflow.DeidentifyFileRequestV2, + requestOptions?: FilesV2.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/detect/deidentify/file", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.DeidentifyFileResponseV2, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/detect/deidentify/file."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Re-identifies tokens in a file.

Requires the Detect Invoker, Detect Manager, or Vault Owner role. Re-identifying `vault_token` values also requires a policy that separately grants the Detokenization action, unless the caller has the Vault Owner role.

Note: V2 API is in beta. + * + * @param {Skyflow.ReidentifyFileRequestV2} request + * @param {FilesV2.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.filesV2.reidentifyFileV2({ + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "txt", + * vaultId: "$VAULT_ID", + * redactionLevel: [{ + * entityName: "EMAIL_ADDRESS", + * redactionType: "MASKED" + * }] + * }) + * + * @example + * await client.filesV2.reidentifyFileV2({ + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "txt", + * vaultId: "$VAULT_ID", + * redactionLevel: [{ + * tokenGroupName: "personTokenGroup", + * redactionPattern: "personMaskPattern" + * }] + * }) + */ + public reidentifyFileV2( + request: Skyflow.ReidentifyFileRequestV2, + requestOptions?: FilesV2.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__reidentifyFileV2(request, requestOptions)); + } + + private async __reidentifyFileV2( + request: Skyflow.ReidentifyFileRequestV2, + requestOptions?: FilesV2.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/detect/reidentify/file", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.ReidentifyFileResponseV2, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/detect/reidentify/file."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Returns the status of a detect run.

Note: V2 API is in beta. + * + * @param {string} runId + * @param {Skyflow.GetRunV2Request} request + * @param {FilesV2.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.filesV2.getRunV2("runId", { + * vaultId: "vaultId" + * }) + */ + public getRunV2( + runId: string, + request: Skyflow.GetRunV2Request, + requestOptions?: FilesV2.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__getRunV2(runId, request, requestOptions)); + } + + private async __getRunV2( + runId: string, + request: Skyflow.GetRunV2Request, + requestOptions?: FilesV2.RequestOptions, + ): Promise> { + const { vaultId } = request; + const _queryParams: Record = {}; + _queryParams["vaultId"] = vaultId; + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + `v2/detect/runs/${encodeURIComponent(runId)}`, + ), + method: "GET", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + queryParameters: _queryParams, + requestType: "json", + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.DetectRunsResponseV2, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling GET /v2/detect/runs/{runId}."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/common/src/_generated_/rest/api/resources/filesV2/client/index.ts b/common/src/_generated_/rest/api/resources/filesV2/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/common/src/_generated_/rest/api/resources/filesV2/client/requests/DeidentifyFileRequestV2.ts b/common/src/_generated_/rest/api/resources/filesV2/client/requests/DeidentifyFileRequestV2.ts new file mode 100644 index 00000000..cdc7810b --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/client/requests/DeidentifyFileRequestV2.ts @@ -0,0 +1,71 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "pdf", + * configuration: { + * vaultId: "$VAULT_ID", + * detect: { + * entities: [{ + * entityType: "NAME", + * deidentificationType: "VAULT_TOKEN", + * destination: "persons.name" + * }, { + * entityType: "DOB", + * deidentificationType: "ENTITY_UNIQUE_COUNTER", + * transformation: { + * shiftDates: { + * minDays: 10, + * maxDays: 30 + * } + * } + * }, { + * entityType: "ALL", + * deidentificationType: "ENTITY_UNIQUE_COUNTER" + * }], + * restrict: ["restrict"], + * skip: ["skip"], + * customTypes: [{ + * label: "EMPLOYEE_ID", + * match: ["EMP-[0-9]{6}"] + * }], + * returnEntities: "EXCLUDE_SENSITIVE_DATA" + * }, + * media: { + * document: { + * pdf: { + * processingMode: "TEXT_LAYER", + * density: 200, + * maxResolution: 3000 + * } + * } + * } + * } + * } + * + * @example + * { + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "pdf", + * configurationId: "$CONFIGURATION_ID" + * } + */ +export interface DeidentifyFileRequestV2 { + /** Data source of the input file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. */ + dataSource: Skyflow.DeidentifyFileRequestV2DataSource; + /** File data corresponding to the specified `dataSource` type. */ + value: string; + /** Format of the input file. The `VAULT_TOKEN` de-identification type is supported only for `txt`, `json`, `csv`, `xml`, `jsonl`, `docx`, `xlsx`, and `pptx` files. */ + dataFormat?: Skyflow.DeidentifyFileRequestV2DataFormat; + /** ID of the Detect configuration to use for de-identification. */ + configurationId?: string; + configuration?: Skyflow.DetectConfigV2; +} diff --git a/common/src/_generated_/rest/api/resources/filesV2/client/requests/GetRunV2Request.ts b/common/src/_generated_/rest/api/resources/filesV2/client/requests/GetRunV2Request.ts new file mode 100644 index 00000000..5ae7011c --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/client/requests/GetRunV2Request.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * { + * vaultId: "vaultId" + * } + */ +export interface GetRunV2Request { + vaultId: string; +} diff --git a/common/src/_generated_/rest/api/resources/filesV2/client/requests/ReidentifyFileRequestV2.ts b/common/src/_generated_/rest/api/resources/filesV2/client/requests/ReidentifyFileRequestV2.ts new file mode 100644 index 00000000..f309dd32 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/client/requests/ReidentifyFileRequestV2.ts @@ -0,0 +1,43 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "txt", + * vaultId: "$VAULT_ID", + * redactionLevel: [{ + * entityName: "EMAIL_ADDRESS", + * redactionType: "MASKED" + * }] + * } + * + * @example + * { + * dataSource: "BASE64", + * value: "$BASE64_DATA", + * dataFormat: "txt", + * vaultId: "$VAULT_ID", + * redactionLevel: [{ + * tokenGroupName: "personTokenGroup", + * redactionPattern: "personMaskPattern" + * }] + * } + */ +export interface ReidentifyFileRequestV2 { + /** Data source of the input file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. */ + dataSource: Skyflow.ReidentifyFileRequestV2DataSource; + /** File data corresponding to the specified `dataSource` type. */ + value: string; + /** Format of the input file. Limited to the formats that support the `VAULT_TOKEN` de-identification type. */ + dataFormat?: Skyflow.ReidentifyFileRequestV2DataFormat; + /** ID of the vault used for de-identification. */ + vaultId: string; + /** Replacement patterns or types applied to entity types or token groups during re-identification. Specifies a replacement pattern/type and the token group or entity type it applies to during re-identification. Set `tokenGroupName` for FlowVault and `entityName` for SkyVault. Similarly, set `redactionType` for FlowVault and `redactionPattern` for SkyVaultto specify the replacement. */ + redactionLevel?: Skyflow.RedactionLevel[]; +} diff --git a/common/src/_generated_/rest/api/resources/filesV2/client/requests/index.ts b/common/src/_generated_/rest/api/resources/filesV2/client/requests/index.ts new file mode 100644 index 00000000..46dd4fbf --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/client/requests/index.ts @@ -0,0 +1,3 @@ +export { type DeidentifyFileRequestV2 } from "./DeidentifyFileRequestV2"; +export { type ReidentifyFileRequestV2 } from "./ReidentifyFileRequestV2"; +export { type GetRunV2Request } from "./GetRunV2Request"; diff --git a/common/src/_generated_/rest/api/resources/filesV2/index.ts b/common/src/_generated_/rest/api/resources/filesV2/index.ts new file mode 100644 index 00000000..c9240f83 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/index.ts @@ -0,0 +1,2 @@ +export * from "./types"; +export * from "./client"; diff --git a/common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataFormat.ts b/common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataFormat.ts new file mode 100644 index 00000000..02a62aa5 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataFormat.ts @@ -0,0 +1,56 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Format of the input file. The `VAULT_TOKEN` de-identification type is supported only for `txt`, `json`, `csv`, `xml`, `jsonl`, `docx`, `xlsx`, and `pptx` files. + */ +export type DeidentifyFileRequestV2DataFormat = + | "mp3" + | "wav" + | "pdf" + | "txt" + | "csv" + | "json" + | "jpg" + | "jpeg" + | "tif" + | "tiff" + | "png" + | "bmp" + | "xls" + | "xlsx" + | "doc" + | "docx" + | "ppt" + | "pptx" + | "xml" + | "dcm" + | "jsonl" + | "zip" + | "gif"; +export const DeidentifyFileRequestV2DataFormat = { + Mp3: "mp3", + Wav: "wav", + Pdf: "pdf", + Txt: "txt", + Csv: "csv", + Json: "json", + Jpg: "jpg", + Jpeg: "jpeg", + Tif: "tif", + Tiff: "tiff", + Png: "png", + Bmp: "bmp", + Xls: "xls", + Xlsx: "xlsx", + Doc: "doc", + Docx: "docx", + Ppt: "ppt", + Pptx: "pptx", + Xml: "xml", + Dcm: "dcm", + Jsonl: "jsonl", + Zip: "zip", + Gif: "gif", +} as const; diff --git a/common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataSource.ts b/common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataSource.ts new file mode 100644 index 00000000..d0dae9b4 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataSource.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Data source of the input file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. + */ +export type DeidentifyFileRequestV2DataSource = "BASE64" | "SKYFLOW_ID" | "PRESIGNED_URL"; +export const DeidentifyFileRequestV2DataSource = { + Base64: "BASE64", + SkyflowId: "SKYFLOW_ID", + PresignedUrl: "PRESIGNED_URL", +} as const; diff --git a/common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataFormat.ts b/common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataFormat.ts new file mode 100644 index 00000000..ca2ff64a --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataFormat.ts @@ -0,0 +1,18 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Format of the input file. Limited to the formats that support the `VAULT_TOKEN` de-identification type. + */ +export type ReidentifyFileRequestV2DataFormat = "txt" | "csv" | "json" | "xlsx" | "docx" | "pptx" | "xml" | "jsonl"; +export const ReidentifyFileRequestV2DataFormat = { + Txt: "txt", + Csv: "csv", + Json: "json", + Xlsx: "xlsx", + Docx: "docx", + Pptx: "pptx", + Xml: "xml", + Jsonl: "jsonl", +} as const; diff --git a/common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataSource.ts b/common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataSource.ts new file mode 100644 index 00000000..7542ba08 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataSource.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Data source of the input file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. + */ +export type ReidentifyFileRequestV2DataSource = "BASE64" | "SKYFLOW_ID" | "PRESIGNED_URL"; +export const ReidentifyFileRequestV2DataSource = { + Base64: "BASE64", + SkyflowId: "SKYFLOW_ID", + PresignedUrl: "PRESIGNED_URL", +} as const; diff --git a/common/src/_generated_/rest/api/resources/filesV2/types/index.ts b/common/src/_generated_/rest/api/resources/filesV2/types/index.ts new file mode 100644 index 00000000..90f61c75 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/filesV2/types/index.ts @@ -0,0 +1,4 @@ +export * from "./DeidentifyFileRequestV2DataSource"; +export * from "./DeidentifyFileRequestV2DataFormat"; +export * from "./ReidentifyFileRequestV2DataSource"; +export * from "./ReidentifyFileRequestV2DataFormat"; diff --git a/common/src/_generated_/rest/api/resources/guardrailsV2/client/Client.ts b/common/src/_generated_/rest/api/resources/guardrailsV2/client/Client.ts new file mode 100644 index 00000000..9d931c6c --- /dev/null +++ b/common/src/_generated_/rest/api/resources/guardrailsV2/client/Client.ts @@ -0,0 +1,144 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace GuardrailsV2 { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class GuardrailsV2 { + constructor(protected readonly _options: GuardrailsV2.Options = {}) {} + + /** + * Preserve safety and compliance with usage policies.

Note: V2 API is in beta. + * + * @param {Skyflow.DetectGuardrailsRequestV2} request + * @param {GuardrailsV2.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.guardrailsV2.checkGuardrailsV2({ + * text: "I love to play cricket.", + * checkToxicity: true, + * denyTopics: ["sports"], + * vaultId: "$VAULT_ID" + * }) + */ + public checkGuardrailsV2( + request: Skyflow.DetectGuardrailsRequestV2, + requestOptions?: GuardrailsV2.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__checkGuardrailsV2(request, requestOptions)); + } + + private async __checkGuardrailsV2( + request: Skyflow.DetectGuardrailsRequestV2, + requestOptions?: GuardrailsV2.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/detect/guardrails", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.DetectGuardrailsResponseV2, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v2/detect/guardrails."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/common/src/_generated_/rest/api/resources/guardrailsV2/client/index.ts b/common/src/_generated_/rest/api/resources/guardrailsV2/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/guardrailsV2/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/DetectGuardrailsRequestV2.ts b/common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/DetectGuardrailsRequestV2.ts new file mode 100644 index 00000000..90f66196 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/DetectGuardrailsRequestV2.ts @@ -0,0 +1,23 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * { + * text: "I love to play cricket.", + * checkToxicity: true, + * denyTopics: ["sports"], + * vaultId: "$VAULT_ID" + * } + */ +export interface DetectGuardrailsRequestV2 { + /** Text to check against guardrails. */ + text: string; + /** If `true`, checks for toxicity in the text. */ + checkToxicity?: boolean; + /** List of topics to deny. */ + denyTopics?: string[]; + /** ID of the vault. */ + vaultId: string; +} diff --git a/common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/index.ts b/common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/index.ts new file mode 100644 index 00000000..a87d2c44 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/guardrailsV2/client/requests/index.ts @@ -0,0 +1 @@ +export { type DetectGuardrailsRequestV2 } from "./DetectGuardrailsRequestV2"; diff --git a/common/src/_generated_/rest/api/resources/guardrailsV2/index.ts b/common/src/_generated_/rest/api/resources/guardrailsV2/index.ts new file mode 100644 index 00000000..5ec76921 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/guardrailsV2/index.ts @@ -0,0 +1 @@ +export * from "./client"; diff --git a/common/src/_generated_/rest/api/resources/index.ts b/common/src/_generated_/rest/api/resources/index.ts index d933c827..1aba4258 100644 --- a/common/src/_generated_/rest/api/resources/index.ts +++ b/common/src/_generated_/rest/api/resources/index.ts @@ -1,2 +1,11 @@ +export * as filesV2 from "./filesV2"; +export * from "./filesV2/types"; export * as authentication from "./authentication"; +export * as tokenExchange from "./tokenExchange"; +export * as stringsV2 from "./stringsV2"; +export * as guardrailsV2 from "./guardrailsV2"; export * from "./authentication/client/requests"; +export * from "./tokenExchange/client/requests"; +export * from "./filesV2/client/requests"; +export * from "./stringsV2/client/requests"; +export * from "./guardrailsV2/client/requests"; diff --git a/common/src/_generated_/rest/api/resources/stringsV2/client/Client.ts b/common/src/_generated_/rest/api/resources/stringsV2/client/Client.ts new file mode 100644 index 00000000..efa23fb6 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/stringsV2/client/Client.ts @@ -0,0 +1,293 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace StringsV2 { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class StringsV2 { + constructor(protected readonly _options: StringsV2.Options = {}) {} + + /** + * De-identifies sensitive data from a string.

Note: V2 API is in beta. + * + * @param {Skyflow.DeidentifyStringRequestV2} request + * @param {StringsV2.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.stringsV2.deidentifyStringV2({ + * text: "My name is John Doe, and my email is johndoe@acme.com.", + * configuration: { + * vaultId: "$VAULT_ID", + * detect: { + * entities: [{ + * entityType: "NAME", + * deidentificationType: "VAULT_TOKEN", + * destination: "persons.name" + * }, { + * entityType: "DOB", + * deidentificationType: "ENTITY_UNIQUE_COUNTER", + * transformation: { + * shiftDates: { + * minDays: 10, + * maxDays: 30 + * } + * } + * }, { + * entityType: "ALL", + * deidentificationType: "ENTITY_UNIQUE_COUNTER" + * }], + * restrict: ["restrict"], + * skip: ["skip"], + * customTypes: [{ + * label: "EMPLOYEE_ID", + * match: ["EMP-[0-9]{6}"] + * }], + * returnEntities: "EXCLUDE_SENSITIVE_DATA" + * } + * } + * }) + * + * @example + * await client.stringsV2.deidentifyStringV2({ + * text: "My name is John Doe, and my email is johndoe@acme.com.", + * configurationId: "$CONFIGURATION_ID" + * }) + */ + public deidentifyStringV2( + request: Skyflow.DeidentifyStringRequestV2, + requestOptions?: StringsV2.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__deidentifyStringV2(request, requestOptions)); + } + + private async __deidentifyStringV2( + request: Skyflow.DeidentifyStringRequestV2, + requestOptions?: StringsV2.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/detect/deidentify/string", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.DeidentifyStringResponseV2, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError( + "Timeout exceeded when calling POST /v2/detect/deidentify/string.", + ); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + /** + * Re-identifies tokens in a string.

Requires the Detect Invoker, Detect Manager, or Vault Owner role. Re-identifying `vault_token` values also requires a policy that separately grants the Detokenization action, unless the caller has the Vault Owner role.

Note: V2 API is in beta. + * + * @param {Skyflow.ReidentifyStringRequestV2} request + * @param {StringsV2.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.stringsV2.reidentifyStringV2({ + * vaultId: "$VAULT_ID", + * text: "My name is [NAME_1], and my email is [EMAIL_ADDRESS_1].", + * redactionLevel: [{ + * entityName: "EMAIL_ADDRESS", + * redactionType: "MASKED" + * }] + * }) + * + * @example + * await client.stringsV2.reidentifyStringV2({ + * vaultId: "$VAULT_ID", + * text: "My name is [NAME_1], and my email is [EMAIL_ADDRESS_1].", + * redactionLevel: [{ + * tokenGroupName: "personTokenGroup", + * redactionPattern: "personMaskPattern" + * }] + * }) + */ + public reidentifyStringV2( + request: Skyflow.ReidentifyStringRequestV2, + requestOptions?: StringsV2.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__reidentifyStringV2(request, requestOptions)); + } + + private async __reidentifyStringV2( + request: Skyflow.ReidentifyStringRequestV2, + requestOptions?: StringsV2.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v2/detect/reidentify/string", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.ReidentifyStringResponseV2, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError( + "Timeout exceeded when calling POST /v2/detect/reidentify/string.", + ); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/common/src/_generated_/rest/api/resources/stringsV2/client/index.ts b/common/src/_generated_/rest/api/resources/stringsV2/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/stringsV2/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/common/src/_generated_/rest/api/resources/stringsV2/client/requests/DeidentifyStringRequestV2.ts b/common/src/_generated_/rest/api/resources/stringsV2/client/requests/DeidentifyStringRequestV2.ts new file mode 100644 index 00000000..6498f1a9 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/stringsV2/client/requests/DeidentifyStringRequestV2.ts @@ -0,0 +1,54 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * text: "My name is John Doe, and my email is johndoe@acme.com.", + * configuration: { + * vaultId: "$VAULT_ID", + * detect: { + * entities: [{ + * entityType: "NAME", + * deidentificationType: "VAULT_TOKEN", + * destination: "persons.name" + * }, { + * entityType: "DOB", + * deidentificationType: "ENTITY_UNIQUE_COUNTER", + * transformation: { + * shiftDates: { + * minDays: 10, + * maxDays: 30 + * } + * } + * }, { + * entityType: "ALL", + * deidentificationType: "ENTITY_UNIQUE_COUNTER" + * }], + * restrict: ["restrict"], + * skip: ["skip"], + * customTypes: [{ + * label: "EMPLOYEE_ID", + * match: ["EMP-[0-9]{6}"] + * }], + * returnEntities: "EXCLUDE_SENSITIVE_DATA" + * } + * } + * } + * + * @example + * { + * text: "My name is John Doe, and my email is johndoe@acme.com.", + * configurationId: "$CONFIGURATION_ID" + * } + */ +export interface DeidentifyStringRequestV2 { + /** Text to deidentify. */ + text: string; + /** ID of the Detect configuration to use for de-identification. */ + configurationId?: string; + configuration?: Skyflow.DetectConfigV2; +} diff --git a/common/src/_generated_/rest/api/resources/stringsV2/client/requests/ReidentifyStringRequestV2.ts b/common/src/_generated_/rest/api/resources/stringsV2/client/requests/ReidentifyStringRequestV2.ts new file mode 100644 index 00000000..12c2b28c --- /dev/null +++ b/common/src/_generated_/rest/api/resources/stringsV2/client/requests/ReidentifyStringRequestV2.ts @@ -0,0 +1,35 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../../../../index"; + +/** + * @example + * { + * vaultId: "$VAULT_ID", + * text: "My name is [NAME_1], and my email is [EMAIL_ADDRESS_1].", + * redactionLevel: [{ + * entityName: "EMAIL_ADDRESS", + * redactionType: "MASKED" + * }] + * } + * + * @example + * { + * vaultId: "$VAULT_ID", + * text: "My name is [NAME_1], and my email is [EMAIL_ADDRESS_1].", + * redactionLevel: [{ + * tokenGroupName: "personTokenGroup", + * redactionPattern: "personMaskPattern" + * }] + * } + */ +export interface ReidentifyStringRequestV2 { + /** ID of the vault used for re-identification. Requires the Detect Invoker, Detect Manager, or Vault Owner role for this vault. Re-identifying `vault_token` values also requires a role or policy that separately grants the Detokenization action, unless the caller has the Vault Owner role. */ + vaultId: string; + /** Text to reidentify. */ + text: string; + /** Replacement patterns or types applied to entity types or token groups during re-identification. Specifies a replacement pattern/type and the token group or entity type it applies to during re-identification. Set exactly one of `tokenGroupName` or `entityName`. Use `tokenGroupName` for FlowVault and `entityName` for SkyVault. Similarly, set exactly one of `redactionType` or `redactionPattern` to specify the replacement. */ + redactionLevel?: Skyflow.RedactionLevel[]; +} diff --git a/common/src/_generated_/rest/api/resources/stringsV2/client/requests/index.ts b/common/src/_generated_/rest/api/resources/stringsV2/client/requests/index.ts new file mode 100644 index 00000000..bee01c0d --- /dev/null +++ b/common/src/_generated_/rest/api/resources/stringsV2/client/requests/index.ts @@ -0,0 +1,2 @@ +export { type DeidentifyStringRequestV2 } from "./DeidentifyStringRequestV2"; +export { type ReidentifyStringRequestV2 } from "./ReidentifyStringRequestV2"; diff --git a/common/src/_generated_/rest/api/resources/stringsV2/index.ts b/common/src/_generated_/rest/api/resources/stringsV2/index.ts new file mode 100644 index 00000000..5ec76921 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/stringsV2/index.ts @@ -0,0 +1 @@ +export * from "./client"; diff --git a/common/src/_generated_/rest/api/resources/tokenExchange/client/Client.ts b/common/src/_generated_/rest/api/resources/tokenExchange/client/Client.ts new file mode 100644 index 00000000..f59af95b --- /dev/null +++ b/common/src/_generated_/rest/api/resources/tokenExchange/client/Client.ts @@ -0,0 +1,150 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as environments from "../../../../environments"; +import * as core from "../../../../core"; +import * as Skyflow from "../../../index"; +import urlJoin from "url-join"; +import * as errors from "../../../../errors/index"; + +export declare namespace TokenExchange { + export interface Options { + environment?: core.Supplier; + /** Specify a custom URL to connect the client to. */ + baseUrl?: core.Supplier; + token?: core.Supplier; + fetcher?: core.FetchFunction; + } + + export interface RequestOptions { + /** The maximum time to wait for a response in seconds. */ + timeoutInSeconds?: number; + /** The number of times to retry the request. Defaults to 2. */ + maxRetries?: number; + /** A hook to abort the request. */ + abortSignal?: AbortSignal; + /** Additional headers to include in the request. */ + headers?: Record; + } +} + +export class TokenExchange { + constructor(protected readonly _options: TokenExchange.Options = {}) {} + + /** + * Generates a Bearer Token to authenticate with Skyflow. + * + * @param {Skyflow.GetStsTokenRequest} request + * @param {TokenExchange.RequestOptions} requestOptions - Request-specific configuration. + * + * @throws {@link Skyflow.BadRequestError} + * @throws {@link Skyflow.UnauthorizedError} + * @throws {@link Skyflow.NotFoundError} + * @throws {@link Skyflow.InternalServerError} + * + * @example + * await client.tokenExchange.getStsToken({ + * grant_type: "grant_type", + * subject_token: "subject_token", + * subject_token_type: "subject_token_type", + * service_account_id: "service_account_id" + * }) + */ + public getStsToken( + request: Skyflow.GetStsTokenRequest, + requestOptions?: TokenExchange.RequestOptions, + ): core.HttpResponsePromise { + return core.HttpResponsePromise.fromPromise(this.__getStsToken(request, requestOptions)); + } + + private async __getStsToken( + request: Skyflow.GetStsTokenRequest, + requestOptions?: TokenExchange.RequestOptions, + ): Promise> { + const _response = await (this._options.fetcher ?? core.fetcher)({ + url: urlJoin( + (await core.Supplier.get(this._options.baseUrl)) ?? + (await core.Supplier.get(this._options.environment)) ?? + environments.SkyflowEnvironment.Production, + "v1/auth/sts/token", + ), + method: "POST", + headers: { + Authorization: await this._getAuthorizationHeader(), + "X-Fern-Language": "JavaScript", + "X-Fern-SDK-Name": "skyflow", + "X-Fern-SDK-Version": "1.0.21", + "User-Agent": "skyflow/1.0.21", + "X-Fern-Runtime": core.RUNTIME.type, + "X-Fern-Runtime-Version": core.RUNTIME.version, + ...requestOptions?.headers, + }, + contentType: "application/json", + requestType: "json", + body: request, + timeoutMs: requestOptions?.timeoutInSeconds != null ? requestOptions.timeoutInSeconds * 1000 : 60000, + maxRetries: requestOptions?.maxRetries, + abortSignal: requestOptions?.abortSignal, + }); + if (_response.ok) { + return { data: _response.body as Skyflow.GetStsTokenResponse, rawResponse: _response.rawResponse }; + } + + if (_response.error.reason === "status-code") { + switch (_response.error.statusCode) { + case 400: + throw new Skyflow.BadRequestError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 401: + throw new Skyflow.UnauthorizedError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 404: + throw new Skyflow.NotFoundError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + case 500: + throw new Skyflow.InternalServerError( + _response.error.body as Skyflow.ErrorResponse, + _response.rawResponse, + ); + default: + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.body, + rawResponse: _response.rawResponse, + }); + } + } + + switch (_response.error.reason) { + case "non-json": + throw new errors.SkyflowError({ + statusCode: _response.error.statusCode, + body: _response.error.rawBody, + rawResponse: _response.rawResponse, + }); + case "timeout": + throw new errors.SkyflowTimeoutError("Timeout exceeded when calling POST /v1/auth/sts/token."); + case "unknown": + throw new errors.SkyflowError({ + message: _response.error.errorMessage, + rawResponse: _response.rawResponse, + }); + } + } + + protected async _getAuthorizationHeader(): Promise { + const bearer = await core.Supplier.get(this._options.token); + if (bearer != null) { + return `Bearer ${bearer}`; + } + + return undefined; + } +} diff --git a/common/src/_generated_/rest/api/resources/tokenExchange/client/index.ts b/common/src/_generated_/rest/api/resources/tokenExchange/client/index.ts new file mode 100644 index 00000000..415726b7 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/tokenExchange/client/index.ts @@ -0,0 +1 @@ +export * from "./requests"; diff --git a/common/src/_generated_/rest/api/resources/tokenExchange/client/requests/GetStsTokenRequest.ts b/common/src/_generated_/rest/api/resources/tokenExchange/client/requests/GetStsTokenRequest.ts new file mode 100644 index 00000000..dcb35600 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/tokenExchange/client/requests/GetStsTokenRequest.ts @@ -0,0 +1,21 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * @example + * { + * grant_type: "grant_type", + * subject_token: "subject_token", + * subject_token_type: "subject_token_type", + * service_account_id: "service_account_id" + * } + */ +export interface GetStsTokenRequest { + grant_type: string; + /** Subject token. */ + subject_token: string; + /** Subject token type. */ + subject_token_type: string; + service_account_id: string; +} diff --git a/common/src/_generated_/rest/api/resources/tokenExchange/client/requests/index.ts b/common/src/_generated_/rest/api/resources/tokenExchange/client/requests/index.ts new file mode 100644 index 00000000..9e5b9fa4 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/tokenExchange/client/requests/index.ts @@ -0,0 +1 @@ +export { type GetStsTokenRequest } from "./GetStsTokenRequest"; diff --git a/common/src/_generated_/rest/api/resources/tokenExchange/index.ts b/common/src/_generated_/rest/api/resources/tokenExchange/index.ts new file mode 100644 index 00000000..5ec76921 --- /dev/null +++ b/common/src/_generated_/rest/api/resources/tokenExchange/index.ts @@ -0,0 +1 @@ +export * from "./client"; diff --git a/common/src/_generated_/rest/api/types/AudioV2.ts b/common/src/_generated_/rest/api/types/AudioV2.ts new file mode 100644 index 00000000..2d7f3926 --- /dev/null +++ b/common/src/_generated_/rest/api/types/AudioV2.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Audio detection and deidentification configuration. + */ +export interface AudioV2 { + /** If `true`, includes processed audio file in the response. Can't be used together with `outputTranscription`. */ + outputProcessedAudio?: boolean; + /** Type of transcription to include in the response. `TRANSCRIPTION`: standard transcript of the audio. `MEDICAL_TRANSCRIPTION`: transcript focused on medical content and terminology. `DIARIZED_TRANSCRIPTION`: speaker-separated transcript that labels each speaker's dialogue. `MEDICAL_DIARIZED_TRANSCRIPTION`: speaker-separated transcript focused on medical content and terminology. `NONE`: no transcription is included. Can't be used together with `outputProcessedAudio`. */ + outputTranscription?: Skyflow.AudioV2OutputTranscription; + bleep?: Skyflow.AudioV2Bleep; +} diff --git a/common/src/_generated_/rest/api/types/AudioV2Bleep.ts b/common/src/_generated_/rest/api/types/AudioV2Bleep.ts new file mode 100644 index 00000000..119f04d5 --- /dev/null +++ b/common/src/_generated_/rest/api/types/AudioV2Bleep.ts @@ -0,0 +1,17 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Configuration for the bleep sound used to mask detected entities in audio. + */ +export interface AudioV2Bleep { + /** Padding added to the beginning of a bleep, in seconds. Range: 0–20. Defaults to 0.5. */ + startPadding?: number; + /** Padding added to the end of a bleep, in seconds. Range: 0–20. Defaults to 0.2. */ + stopPadding?: number; + /** Frequency of the sine wave used for the bleep sound, in Hz. */ + frequency?: number; + /** Relative loudness of the bleep. Positive values increase loudness and negative values decrease it. */ + gain?: number; +} diff --git a/common/src/_generated_/rest/api/types/AudioV2OutputTranscription.ts b/common/src/_generated_/rest/api/types/AudioV2OutputTranscription.ts new file mode 100644 index 00000000..77a445a7 --- /dev/null +++ b/common/src/_generated_/rest/api/types/AudioV2OutputTranscription.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of transcription to include in the response. `TRANSCRIPTION`: standard transcript of the audio. `MEDICAL_TRANSCRIPTION`: transcript focused on medical content and terminology. `DIARIZED_TRANSCRIPTION`: speaker-separated transcript that labels each speaker's dialogue. `MEDICAL_DIARIZED_TRANSCRIPTION`: speaker-separated transcript focused on medical content and terminology. `NONE`: no transcription is included. Can't be used together with `outputProcessedAudio`. + */ +export type AudioV2OutputTranscription = + "NONE" | "TRANSCRIPTION" | "MEDICAL_TRANSCRIPTION" | "DIARIZED_TRANSCRIPTION" | "MEDICAL_DIARIZED_TRANSCRIPTION"; +export const AudioV2OutputTranscription = { + None: "NONE", + Transcription: "TRANSCRIPTION", + MedicalTranscription: "MEDICAL_TRANSCRIPTION", + DiarizedTranscription: "DIARIZED_TRANSCRIPTION", + MedicalDiarizedTranscription: "MEDICAL_DIARIZED_TRANSCRIPTION", +} as const; diff --git a/common/src/_generated_/rest/api/types/DeidentifyFileResponseV2.ts b/common/src/_generated_/rest/api/types/DeidentifyFileResponseV2.ts new file mode 100644 index 00000000..58715383 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DeidentifyFileResponseV2.ts @@ -0,0 +1,11 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Response containing the run ID for tracking the async deidentification job. + */ +export interface DeidentifyFileResponseV2 { + /** Run ID of the deidentification request. */ + runId?: string; +} diff --git a/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2.ts b/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2.ts new file mode 100644 index 00000000..1367e6f4 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Response containing the deidentified string and detected entity metadata. + */ +export interface DeidentifyStringResponseV2 { + /** Deidentified text with sensitive entities replaced. */ + processedText?: string; + /** Detected entities. */ + entities?: Skyflow.DeidentifyStringResponseV2DetectedEntity[]; + metrics?: Skyflow.Metrics; +} diff --git a/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2DetectedEntity.ts b/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2DetectedEntity.ts new file mode 100644 index 00000000..2f191484 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2DetectedEntity.ts @@ -0,0 +1,20 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Detected entities for String + */ +export interface DeidentifyStringResponseV2DetectedEntity { + /** Processed text of the entity. */ + token?: string; + /** Original text of the entity. */ + value?: string; + location?: Skyflow.DeidentifyStringResponseV2EntityLocation; + /** Highest-rated label. */ + entityType?: string; + /** Labels and their scores. */ + entityScores?: Record; +} diff --git a/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2EntityLocation.ts b/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2EntityLocation.ts new file mode 100644 index 00000000..939d56dd --- /dev/null +++ b/common/src/_generated_/rest/api/types/DeidentifyStringResponseV2EntityLocation.ts @@ -0,0 +1,17 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Locations of an entity in the text. + */ +export interface DeidentifyStringResponseV2EntityLocation { + /** Index of the first character of the string in the original text. */ + startIndex?: number; + /** Index of the last character of the string in the original text. */ + endIndex?: number; + /** Index of the first character of the string in the processed text. */ + startIndexProcessed?: number; + /** Index of the last character of the string in the processed text. */ + endIndexProcessed?: number; +} diff --git a/common/src/_generated_/rest/api/types/Detect.ts b/common/src/_generated_/rest/api/types/Detect.ts new file mode 100644 index 00000000..33c027d6 --- /dev/null +++ b/common/src/_generated_/rest/api/types/Detect.ts @@ -0,0 +1,23 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Entity detection and deidentification settings. + */ +export interface Detect { + /** Entities and their deidentification settings. */ + entities?: Skyflow.DetectEntity[]; + /** Object entities and their deidentification settings. */ + objectEntities?: Skyflow.DetectObjectEntity[]; + /** Settings for restricting detected entities. A detected entity is replaced with '[RESTRICTED]' only when a `restrict` pattern matches the entity's full value. If a pattern matches only a substring of the detected entity, the pattern is ignored and the entity is processed according to the configured tokenization and transformation settings. Patterns don't match across entity boundaries. If an entity matches both a `skip` and a `restrict` pattern, `skip` takes precedence and the entity is displayed in plaintext. Note: Vault tokens are currently not supported. */ + restrict?: string[]; + /** Settings for bypassing deidentification on detected entities. A detected entity is displayed in plaintext when a `skip` pattern matches either the entity's full value or any substring of it. Patterns don't match across entity boundaries. If an entity matches both a `skip` and a `restrict` pattern, `skip` takes precedence and the entity is displayed in plaintext. */ + skip?: string[]; + /** Custom entity type definitions. */ + customTypes?: Skyflow.DetectCustomTypes[]; + /** Controls which detected entities are returned in the response. `EXCLUDE_SENSITIVE_DATA`: returns each detected entity's metadata (token, entity type, location, and confidence scores) but omits the original sensitive value. `ALL`: returns each detected entity's metadata and the original sensitive value. `NONE`: doesn't return detected entities. */ + returnEntities?: Skyflow.DetectReturnEntities; +} diff --git a/common/src/_generated_/rest/api/types/DetectConfigV2.ts b/common/src/_generated_/rest/api/types/DetectConfigV2.ts new file mode 100644 index 00000000..fce06674 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectConfigV2.ts @@ -0,0 +1,27 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Configuration for detecting and deidentifying entities in a vault. + */ +export interface DetectConfigV2 { + /** ID of the configuration. */ + ID?: string; + /** Name of the configuration. */ + name?: string; + /** Namespace of the configuration. */ + namespace?: string; + /** Description of the configuration. */ + description?: string; + /** ID of the vault. */ + vaultId: string; + /** Type of the vault. */ + vaultType?: Skyflow.DetectConfigV2VaultType; + detect?: Skyflow.Detect; + /** Mappings for source and deidentified file locations. */ + fileMapping?: Skyflow.FileMapping[]; + media?: Skyflow.Media; +} diff --git a/common/src/_generated_/rest/api/types/DetectConfigV2VaultType.ts b/common/src/_generated_/rest/api/types/DetectConfigV2VaultType.ts new file mode 100644 index 00000000..c5a1dbbc --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectConfigV2VaultType.ts @@ -0,0 +1,14 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of the vault. + */ +export type DetectConfigV2VaultType = "NONE" | "PRIVACYDB" | "FLOWDB" | "FLOWDB_SCHEMALESS"; +export const DetectConfigV2VaultType = { + None: "NONE", + Privacydb: "PRIVACYDB", + Flowdb: "FLOWDB", + FlowdbSchemaless: "FLOWDB_SCHEMALESS", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectCustomTypes.ts b/common/src/_generated_/rest/api/types/DetectCustomTypes.ts new file mode 100644 index 00000000..2bbe89b9 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectCustomTypes.ts @@ -0,0 +1,23 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Custom entity type defined by a label and a list of match values. + */ +export interface DetectCustomTypes { + /** Label of the custom type. Used as the entity's identity and default token prefix. */ + label?: string; + /** Regular expressions to replace with '[]' enclosed in square brackets. Expressions must match the entirety of a detected entity, not just a substring, for the entity to be redacted. Expressions don't match across entity boundaries. If a string or entity matches both `skip` and `customType.label`, the entity is displayed in plaintext. If a string is detected as an entity and a `customType.match` pattern matches the entire detected entity, the entity is replaced with '[]'. If a string is detected as an entity but a `customType.match` pattern only matches a substring of it, the `customType.match` pattern is ignored, and the entity is processed according to the specified tokenization and transformation settings. */ + match?: string[]; + /** Type of deidentification to apply to matches of this custom type. */ + deidentificationType?: Skyflow.DetectCustomTypesDeidentificationType; + /** Vault destination where matches of this custom type are stored. Required when `deidentificationType` is `VAULT_TOKEN`. For vaults with a schema, specify the table and column in `table.column` format (for example, `persons.product`). For schemaless vaults, specify the token group name (for example, `productTokenGroup`). */ + destination?: string; + /** Whether to include the label as a prefix in the generated token. Only applies when `deidentificationType` is `VAULT_TOKEN`. `NO` returns the raw vault token, and the value can't be re-identified later. */ + includePrefix?: Skyflow.DetectCustomTypesIncludePrefix; + /** Overrides the default entity-type prefix used in the generated token (for example, `NAAM` instead of `NAME`, producing `[NAAM_]`). Can't be combined with `includePrefix: NO`. */ + prefixOverride?: string; +} diff --git a/common/src/_generated_/rest/api/types/DetectCustomTypesDeidentificationType.ts b/common/src/_generated_/rest/api/types/DetectCustomTypesDeidentificationType.ts new file mode 100644 index 00000000..bc09d855 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectCustomTypesDeidentificationType.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of deidentification to apply to matches of this custom type. + */ +export type DetectCustomTypesDeidentificationType = + "UNKNOWN" | "ENTITY_UNIQUE_COUNTER" | "ENTITY_ONLY" | "VAULT_TOKEN" | "MASK"; +export const DetectCustomTypesDeidentificationType = { + Unknown: "UNKNOWN", + EntityUniqueCounter: "ENTITY_UNIQUE_COUNTER", + EntityOnly: "ENTITY_ONLY", + VaultToken: "VAULT_TOKEN", + Mask: "MASK", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectCustomTypesIncludePrefix.ts b/common/src/_generated_/rest/api/types/DetectCustomTypesIncludePrefix.ts new file mode 100644 index 00000000..6f4006fc --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectCustomTypesIncludePrefix.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Whether to include the label as a prefix in the generated token. Only applies when `deidentificationType` is `VAULT_TOKEN`. `NO` returns the raw vault token, and the value can't be re-identified later. + */ +export type DetectCustomTypesIncludePrefix = "YES" | "NO"; +export const DetectCustomTypesIncludePrefix = { + Yes: "YES", + No: "NO", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectEntity.ts b/common/src/_generated_/rest/api/types/DetectEntity.ts new file mode 100644 index 00000000..dddb0d87 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectEntity.ts @@ -0,0 +1,22 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Entity detection and deidentification configuration. + */ +export interface DetectEntity { + /** Type of entity to detect. */ + entityType?: Skyflow.DetectEntityEntityType; + /** Type of deidentification to apply. `VAULT_TOKEN` is only supported for the following file formats: `txt`, `json`, `csv`, `xml`, `jsonl`, `docx`, `xlsx`, `pptx`. */ + deidentificationType?: Skyflow.DetectEntityDeidentificationType; + /** Vault destination where detected values of this entity type are stored. For vaults with a schema, specify the table and column in `table.column` format (for example, `persons.name`). For schemaless vaults, specify the token group name (for example, `personTokenGroup`). */ + destination?: string; + transformation?: Skyflow.EntityTransformation; + /** Whether to include the label as a prefix in the generated token. Only applies when `deidentificationType` is `VAULT_TOKEN`. `NO` returns the raw vault token, and the value can't be re-identified later. */ + includePrefix?: Skyflow.DetectEntityIncludePrefix; + /** Overrides the default entity-type prefix used in the generated token (for example, `NAAM` instead of `NAME`, producing `[NAAM_]`). Can't be combined with `includePrefix: NO`. */ + prefixOverride?: string; +} diff --git a/common/src/_generated_/rest/api/types/DetectEntityDeidentificationType.ts b/common/src/_generated_/rest/api/types/DetectEntityDeidentificationType.ts new file mode 100644 index 00000000..723ebfca --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectEntityDeidentificationType.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of deidentification to apply. `VAULT_TOKEN` is only supported for the following file formats: `txt`, `json`, `csv`, `xml`, `jsonl`, `docx`, `xlsx`, `pptx`. + */ +export type DetectEntityDeidentificationType = + "UNKNOWN" | "ENTITY_UNIQUE_COUNTER" | "ENTITY_ONLY" | "VAULT_TOKEN" | "MASK"; +export const DetectEntityDeidentificationType = { + Unknown: "UNKNOWN", + EntityUniqueCounter: "ENTITY_UNIQUE_COUNTER", + EntityOnly: "ENTITY_ONLY", + VaultToken: "VAULT_TOKEN", + Mask: "MASK", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectEntityEntityType.ts b/common/src/_generated_/rest/api/types/DetectEntityEntityType.ts new file mode 100644 index 00000000..ac7fe659 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectEntityEntityType.ts @@ -0,0 +1,152 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of entity to detect. + */ +export type DetectEntityEntityType = + | "NONE" + | "BANK_ACCOUNT" + | "CREDIT_CARD" + | "CREDIT_CARD_EXPIRATION" + | "CVV" + | "DATE" + | "DATE_INTERVAL" + | "DOB" + | "DRIVER_LICENSE" + | "EMAIL_ADDRESS" + | "HEALTHCARE_NUMBER" + | "IP_ADDRESS" + | "LOCATION" + | "NAME" + | "NUMERICAL_PII" + | "PHONE_NUMBER" + | "SSN" + | "URL" + | "VEHICLE_ID" + | "MEDICAL_CODE" + | "NAME_FAMILY" + | "NAME_GIVEN" + | "ACCOUNT_NUMBER" + | "EVENT" + | "FILENAME" + | "GENDER" + | "LANGUAGE" + | "LOCATION_ADDRESS" + | "LOCATION_CITY" + | "LOCATION_COORDINATE" + | "LOCATION_COUNTRY" + | "LOCATION_STATE" + | "LOCATION_ZIP" + | "MARITAL_STATUS" + | "MONEY" + | "NAME_MEDICAL_PROFESSIONAL" + | "OCCUPATION" + | "ORGANIZATION" + | "ORGANIZATION_MEDICAL_FACILITY" + | "ORIGIN" + | "PASSPORT_NUMBER" + | "PASSWORD" + | "PHYSICAL_ATTRIBUTE" + | "POLITICAL_AFFILIATION" + | "RELIGION" + | "TIME" + | "USERNAME" + | "ZODIAC_SIGN" + | "BLOOD_TYPE" + | "CONDITION" + | "DOSE" + | "DRUG" + | "INJURY" + | "MEDICAL_PROCESS" + | "STATISTICS" + | "ROUTING_NUMBER" + | "CORPORATE_ACTION" + | "FINANCIAL_METRIC" + | "PRODUCT" + | "TREND" + | "DURATION" + | "LOCATION_ADDRESS_STREET" + | "AGE" + | "SEXUALITY" + | "EFFECT" + | "PROJECT" + | "ORGANIZATION_ID" + | "DAY" + | "MONTH" + | "YEAR" + | "ALL"; +export const DetectEntityEntityType = { + None: "NONE", + BankAccount: "BANK_ACCOUNT", + CreditCard: "CREDIT_CARD", + CreditCardExpiration: "CREDIT_CARD_EXPIRATION", + Cvv: "CVV", + Date: "DATE", + DateInterval: "DATE_INTERVAL", + Dob: "DOB", + DriverLicense: "DRIVER_LICENSE", + EmailAddress: "EMAIL_ADDRESS", + HealthcareNumber: "HEALTHCARE_NUMBER", + IpAddress: "IP_ADDRESS", + Location: "LOCATION", + Name: "NAME", + NumericalPii: "NUMERICAL_PII", + PhoneNumber: "PHONE_NUMBER", + Ssn: "SSN", + Url: "URL", + VehicleId: "VEHICLE_ID", + MedicalCode: "MEDICAL_CODE", + NameFamily: "NAME_FAMILY", + NameGiven: "NAME_GIVEN", + AccountNumber: "ACCOUNT_NUMBER", + Event: "EVENT", + Filename: "FILENAME", + Gender: "GENDER", + Language: "LANGUAGE", + LocationAddress: "LOCATION_ADDRESS", + LocationCity: "LOCATION_CITY", + LocationCoordinate: "LOCATION_COORDINATE", + LocationCountry: "LOCATION_COUNTRY", + LocationState: "LOCATION_STATE", + LocationZip: "LOCATION_ZIP", + MaritalStatus: "MARITAL_STATUS", + Money: "MONEY", + NameMedicalProfessional: "NAME_MEDICAL_PROFESSIONAL", + Occupation: "OCCUPATION", + Organization: "ORGANIZATION", + OrganizationMedicalFacility: "ORGANIZATION_MEDICAL_FACILITY", + Origin: "ORIGIN", + PassportNumber: "PASSPORT_NUMBER", + Password: "PASSWORD", + PhysicalAttribute: "PHYSICAL_ATTRIBUTE", + PoliticalAffiliation: "POLITICAL_AFFILIATION", + Religion: "RELIGION", + Time: "TIME", + Username: "USERNAME", + ZodiacSign: "ZODIAC_SIGN", + BloodType: "BLOOD_TYPE", + Condition: "CONDITION", + Dose: "DOSE", + Drug: "DRUG", + Injury: "INJURY", + MedicalProcess: "MEDICAL_PROCESS", + Statistics: "STATISTICS", + RoutingNumber: "ROUTING_NUMBER", + CorporateAction: "CORPORATE_ACTION", + FinancialMetric: "FINANCIAL_METRIC", + Product: "PRODUCT", + Trend: "TREND", + Duration: "DURATION", + LocationAddressStreet: "LOCATION_ADDRESS_STREET", + Age: "AGE", + Sexuality: "SEXUALITY", + Effect: "EFFECT", + Project: "PROJECT", + OrganizationId: "ORGANIZATION_ID", + Day: "DAY", + Month: "MONTH", + Year: "YEAR", + All: "ALL", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectEntityIncludePrefix.ts b/common/src/_generated_/rest/api/types/DetectEntityIncludePrefix.ts new file mode 100644 index 00000000..8370a186 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectEntityIncludePrefix.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Whether to include the label as a prefix in the generated token. Only applies when `deidentificationType` is `VAULT_TOKEN`. `NO` returns the raw vault token, and the value can't be re-identified later. + */ +export type DetectEntityIncludePrefix = "YES" | "NO"; +export const DetectEntityIncludePrefix = { + Yes: "YES", + No: "NO", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2.ts b/common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2.ts new file mode 100644 index 00000000..a647a887 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Response object for detect guardrails + */ +export interface DetectGuardrailsResponseV2 { + /** Text that was checked against guardrails. */ + text: string; + /** Whether the text is toxic. */ + toxic?: boolean; + /** Whether the text included a denied topic. */ + deniedTopic?: boolean; + /** Whether the text passed validation. */ + validation: Skyflow.DetectGuardrailsResponseV2Validation; +} diff --git a/common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2Validation.ts b/common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2Validation.ts new file mode 100644 index 00000000..7c837a64 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectGuardrailsResponseV2Validation.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Whether the text passed validation. + */ +export type DetectGuardrailsResponseV2Validation = "FAILED" | "PASSED"; +export const DetectGuardrailsResponseV2Validation = { + Failed: "FAILED", + Passed: "PASSED", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectObjectEntity.ts b/common/src/_generated_/rest/api/types/DetectObjectEntity.ts new file mode 100644 index 00000000..ec6274e8 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectObjectEntity.ts @@ -0,0 +1,15 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Object entity detection and deidentification configuration (e.g. faces, license plates). + */ +export interface DetectObjectEntity { + /** Type of entity to detect. */ + entityType?: Skyflow.DetectObjectEntityEntityType; + /** Type of deidentification to apply. */ + deidentificationType?: Skyflow.DetectObjectEntityDeidentificationType; +} diff --git a/common/src/_generated_/rest/api/types/DetectObjectEntityDeidentificationType.ts b/common/src/_generated_/rest/api/types/DetectObjectEntityDeidentificationType.ts new file mode 100644 index 00000000..64fbca85 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectObjectEntityDeidentificationType.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of deidentification to apply. + */ +export type DetectObjectEntityDeidentificationType = "UNKNOWN" | "REDACT" | "UNREDACT"; +export const DetectObjectEntityDeidentificationType = { + Unknown: "UNKNOWN", + Redact: "REDACT", + Unredact: "UNREDACT", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectObjectEntityEntityType.ts b/common/src/_generated_/rest/api/types/DetectObjectEntityEntityType.ts new file mode 100644 index 00000000..d30db76a --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectObjectEntityEntityType.ts @@ -0,0 +1,16 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of entity to detect. + */ +export type DetectObjectEntityEntityType = "NONE" | "FACE" | "LICENSE_PLATE" | "LOGO" | "SIGNATURE" | "ALL"; +export const DetectObjectEntityEntityType = { + None: "NONE", + Face: "FACE", + LicensePlate: "LICENSE_PLATE", + Logo: "LOGO", + Signature: "SIGNATURE", + All: "ALL", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectReturnEntities.ts b/common/src/_generated_/rest/api/types/DetectReturnEntities.ts new file mode 100644 index 00000000..bf02580a --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectReturnEntities.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Controls which detected entities are returned in the response. `EXCLUDE_SENSITIVE_DATA`: returns each detected entity's metadata (token, entity type, location, and confidence scores) but omits the original sensitive value. `ALL`: returns each detected entity's metadata and the original sensitive value. `NONE`: doesn't return detected entities. + */ +export type DetectReturnEntities = "EXCLUDE_SENSITIVE_DATA" | "ALL" | "NONE"; +export const DetectReturnEntities = { + ExcludeSensitiveData: "EXCLUDE_SENSITIVE_DATA", + All: "ALL", + None: "NONE", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectRunsResponseV2.ts b/common/src/_generated_/rest/api/types/DetectRunsResponseV2.ts new file mode 100644 index 00000000..9829287f --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectRunsResponseV2.ts @@ -0,0 +1,20 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Response containing the status and output of a v2 detect run. + */ +export interface DetectRunsResponseV2 { + /** Status of the detect run. */ + status?: Skyflow.DetectRunsResponseV2Status; + /** Data source of the output file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. */ + outputType?: Skyflow.DetectRunsResponseV2OutputType; + /** Details of output files. */ + output?: Skyflow.FileOutput[]; + /** Status details about the Detect run. */ + message?: string; + metrics?: Skyflow.Metrics; +} diff --git a/common/src/_generated_/rest/api/types/DetectRunsResponseV2OutputType.ts b/common/src/_generated_/rest/api/types/DetectRunsResponseV2OutputType.ts new file mode 100644 index 00000000..a09a432a --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectRunsResponseV2OutputType.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Data source of the output file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. + */ +export type DetectRunsResponseV2OutputType = "BASE64" | "SKYFLOW_ID" | "PRESIGNED_URL"; +export const DetectRunsResponseV2OutputType = { + Base64: "BASE64", + SkyflowId: "SKYFLOW_ID", + PresignedUrl: "PRESIGNED_URL", +} as const; diff --git a/common/src/_generated_/rest/api/types/DetectRunsResponseV2Status.ts b/common/src/_generated_/rest/api/types/DetectRunsResponseV2Status.ts new file mode 100644 index 00000000..84dd469d --- /dev/null +++ b/common/src/_generated_/rest/api/types/DetectRunsResponseV2Status.ts @@ -0,0 +1,15 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Status of the detect run. + */ +export type DetectRunsResponseV2Status = "UNKNOWN" | "FAILED" | "SUCCESS" | "IN_PROGRESS" | "QUEUED"; +export const DetectRunsResponseV2Status = { + Unknown: "UNKNOWN", + Failed: "FAILED", + Success: "SUCCESS", + InProgress: "IN_PROGRESS", + Queued: "QUEUED", +} as const; diff --git a/common/src/_generated_/rest/api/types/DocumentV2.ts b/common/src/_generated_/rest/api/types/DocumentV2.ts new file mode 100644 index 00000000..fb5a3b49 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DocumentV2.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Document detection and deidentification configuration. + */ +export interface DocumentV2 { + pdf?: Skyflow.DocumentV2Pdf; +} diff --git a/common/src/_generated_/rest/api/types/DocumentV2Pdf.ts b/common/src/_generated_/rest/api/types/DocumentV2Pdf.ts new file mode 100644 index 00000000..cf521da5 --- /dev/null +++ b/common/src/_generated_/rest/api/types/DocumentV2Pdf.ts @@ -0,0 +1,17 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Advanced options for processing PDF documents. + */ +export interface DocumentV2Pdf { + /** Processing mode for the PDF file. `TEXT_LAYER`: processes the PDF's native text layer and replaces detected entities with entity tokens (for example, `[NAME_1]`). Fast and accurate for PDFs with selectable text. `OCR`: processes the PDF as an image using optical character recognition and covers detected content with a gray box. Use for scanned or image-based PDFs. Because `OCR` rasterizes each page, the output PDF file size can be significantly larger than the input. */ + processingMode?: Skyflow.DocumentV2PdfProcessingMode; + /** Pixel density at which to process the PDF file. Applies only when `processingMode` is `OCR`. */ + density?: number; + /** Max resolution at which to process the PDF file. Applies only when `processingMode` is `OCR`. */ + maxResolution?: number; +} diff --git a/common/src/_generated_/rest/api/types/DocumentV2PdfProcessingMode.ts b/common/src/_generated_/rest/api/types/DocumentV2PdfProcessingMode.ts new file mode 100644 index 00000000..7a86b09e --- /dev/null +++ b/common/src/_generated_/rest/api/types/DocumentV2PdfProcessingMode.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Processing mode for the PDF file. `TEXT_LAYER`: processes the PDF's native text layer and replaces detected entities with entity tokens (for example, `[NAME_1]`). Fast and accurate for PDFs with selectable text. `OCR`: processes the PDF as an image using optical character recognition and covers detected content with a gray box. Use for scanned or image-based PDFs. Because `OCR` rasterizes each page, the output PDF file size can be significantly larger than the input. + */ +export type DocumentV2PdfProcessingMode = "NONE" | "OCR" | "TEXT_LAYER"; +export const DocumentV2PdfProcessingMode = { + None: "NONE", + Ocr: "OCR", + TextLayer: "TEXT_LAYER", +} as const; diff --git a/common/src/_generated_/rest/api/types/EntityTransformation.ts b/common/src/_generated_/rest/api/types/EntityTransformation.ts new file mode 100644 index 00000000..e1f0d1c5 --- /dev/null +++ b/common/src/_generated_/rest/api/types/EntityTransformation.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Optional transformations to apply to detected entity values. + */ +export interface EntityTransformation { + shiftDates?: Skyflow.TransformationShiftDates; +} diff --git a/common/src/_generated_/rest/api/types/GooglerpcStatus.ts b/common/src/_generated_/rest/api/types/ErrorResponse.ts similarity index 50% rename from common/src/_generated_/rest/api/types/GooglerpcStatus.ts rename to common/src/_generated_/rest/api/types/ErrorResponse.ts index 70e20691..822dd151 100644 --- a/common/src/_generated_/rest/api/types/GooglerpcStatus.ts +++ b/common/src/_generated_/rest/api/types/ErrorResponse.ts @@ -4,8 +4,6 @@ import * as Skyflow from "../index"; -export interface GooglerpcStatus { - code?: number; - message?: string; - details?: Skyflow.ProtobufAny[]; +export interface ErrorResponse { + error: Skyflow.ErrorResponseError; } diff --git a/common/src/_generated_/rest/api/types/ErrorResponseError.ts b/common/src/_generated_/rest/api/types/ErrorResponseError.ts new file mode 100644 index 00000000..68e842c9 --- /dev/null +++ b/common/src/_generated_/rest/api/types/ErrorResponseError.ts @@ -0,0 +1,14 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +export interface ErrorResponseError { + /** gRPC status codes. See https://grpc.io/docs/guides/status-codes. */ + grpc_code: number; + http_code: Skyflow.HttpCode; + http_status: string; + message: string; + details?: Record[]; +} diff --git a/common/src/_generated_/rest/api/types/FileMapping.ts b/common/src/_generated_/rest/api/types/FileMapping.ts new file mode 100644 index 00000000..6c46158f --- /dev/null +++ b/common/src/_generated_/rest/api/types/FileMapping.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Source and destination file paths for a deidentification job. + */ +export interface FileMapping { + /** Location of the source file. */ + source?: string; + /** Location of the target file. */ + destination?: string; + /** Suffix of the output file. For example, with outputFileNameSuffix = "_deidentified", "invoice.pdf" becomes "invoice_deidentified.pdf". */ + outputFileNameSuffix?: string; + /** Location of the entities output, as "table.column". For example, "documents.entities_json". Must be in the same table as source. If unset, the entities output is not stored in the vault. */ + entitiesDestination?: string; + /** Location of the object entities output, as "table.column". For example, "documents.object_entities_json". Must be in the same table as source. If unset, the object entities output is not stored in the vault. */ + objectEntitiesDestination?: string; +} diff --git a/common/src/_generated_/rest/api/types/FileOutput.ts b/common/src/_generated_/rest/api/types/FileOutput.ts new file mode 100644 index 00000000..ee9b94f7 --- /dev/null +++ b/common/src/_generated_/rest/api/types/FileOutput.ts @@ -0,0 +1,17 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Represents a single processed output file along with its type and format. + */ +export interface FileOutput { + /** Value of the processed file. */ + processedFile?: string; + /** Type of the processed file. */ + processedFileType?: Skyflow.FileOutputProcessedFileType; + /** Processed file extension. */ + processedFileExtension?: Skyflow.FileOutputProcessedFileExtension; +} diff --git a/common/src/_generated_/rest/api/types/FileOutputProcessedFileExtension.ts b/common/src/_generated_/rest/api/types/FileOutputProcessedFileExtension.ts new file mode 100644 index 00000000..139f36ee --- /dev/null +++ b/common/src/_generated_/rest/api/types/FileOutputProcessedFileExtension.ts @@ -0,0 +1,56 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Processed file extension. + */ +export type FileOutputProcessedFileExtension = + | "mp3" + | "wav" + | "pdf" + | "txt" + | "csv" + | "json" + | "jpg" + | "jpeg" + | "tif" + | "tiff" + | "png" + | "bmp" + | "xls" + | "xlsx" + | "doc" + | "docx" + | "ppt" + | "pptx" + | "xml" + | "dcm" + | "jsonl" + | "zip" + | "gif"; +export const FileOutputProcessedFileExtension = { + Mp3: "mp3", + Wav: "wav", + Pdf: "pdf", + Txt: "txt", + Csv: "csv", + Json: "json", + Jpg: "jpg", + Jpeg: "jpeg", + Tif: "tif", + Tiff: "tiff", + Png: "png", + Bmp: "bmp", + Xls: "xls", + Xlsx: "xlsx", + Doc: "doc", + Docx: "docx", + Ppt: "ppt", + Pptx: "pptx", + Xml: "xml", + Dcm: "dcm", + Jsonl: "jsonl", + Zip: "zip", + Gif: "gif", +} as const; diff --git a/common/src/_generated_/rest/api/types/FileOutputProcessedFileType.ts b/common/src/_generated_/rest/api/types/FileOutputProcessedFileType.ts new file mode 100644 index 00000000..1e108dad --- /dev/null +++ b/common/src/_generated_/rest/api/types/FileOutputProcessedFileType.ts @@ -0,0 +1,26 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Type of the processed file. + */ +export type FileOutputProcessedFileType = + | "NONE" + | "REDACTED_FILE" + | "REDACTED_TEXT" + | "ENTITIES" + | "REIDENTIFIED_FILE" + | "REDACTED_TRANSCRIPTION" + | "REDACTED_DEEPGRAM_TRANSCRIPTION_DIARIZE_JSON" + | "OBJECT_ENTITIES"; +export const FileOutputProcessedFileType = { + None: "NONE", + RedactedFile: "REDACTED_FILE", + RedactedText: "REDACTED_TEXT", + Entities: "ENTITIES", + ReidentifiedFile: "REIDENTIFIED_FILE", + RedactedTranscription: "REDACTED_TRANSCRIPTION", + RedactedDeepgramTranscriptionDiarizeJson: "REDACTED_DEEPGRAM_TRANSCRIPTION_DIARIZE_JSON", + ObjectEntities: "OBJECT_ENTITIES", +} as const; diff --git a/common/src/_generated_/rest/api/types/V1GetAuthTokenResponse.ts b/common/src/_generated_/rest/api/types/GetAuthTokenResponse.ts similarity index 81% rename from common/src/_generated_/rest/api/types/V1GetAuthTokenResponse.ts rename to common/src/_generated_/rest/api/types/GetAuthTokenResponse.ts index 860942f1..7564b440 100644 --- a/common/src/_generated_/rest/api/types/V1GetAuthTokenResponse.ts +++ b/common/src/_generated_/rest/api/types/GetAuthTokenResponse.ts @@ -2,7 +2,7 @@ * This file was auto-generated by Fern from our API Definition. */ -export interface V1GetAuthTokenResponse { +export interface GetAuthTokenResponse { /** AccessToken. */ accessToken?: string; /** TokenType : Bearer. */ diff --git a/common/src/_generated_/rest/api/types/GetStsTokenResponse.ts b/common/src/_generated_/rest/api/types/GetStsTokenResponse.ts new file mode 100644 index 00000000..f7d13d01 --- /dev/null +++ b/common/src/_generated_/rest/api/types/GetStsTokenResponse.ts @@ -0,0 +1,12 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +export interface GetStsTokenResponse { + /** AccessToken. */ + accessToken?: string; + /** IssuedTokenType : urn:ietf:params:oauth:token-type:jwt. */ + issuedTokenType?: string; + /** TokenType : Bearer. */ + tokenType?: string; +} diff --git a/common/src/_generated_/rest/api/types/HttpCode.ts b/common/src/_generated_/rest/api/types/HttpCode.ts new file mode 100644 index 00000000..e85dbda8 --- /dev/null +++ b/common/src/_generated_/rest/api/types/HttpCode.ts @@ -0,0 +1,8 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * HTTP status codes. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Status. + */ +export type HttpCode = number; diff --git a/common/src/_generated_/rest/api/types/ImageV2.ts b/common/src/_generated_/rest/api/types/ImageV2.ts new file mode 100644 index 00000000..b11a44bf --- /dev/null +++ b/common/src/_generated_/rest/api/types/ImageV2.ts @@ -0,0 +1,17 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Image detection and deidentification configuration. + */ +export interface ImageV2 { + /** If `true`, includes processed image in the output. */ + outputProcessedImage?: boolean; + /** If `true`, includes text detected by OCR in the response. */ + outputOcrText?: boolean; + /** Method used to mask detected entities in the image. `BLUR`: blurs each detected entity so the sensitive content is unreadable while the rest of the image is preserved. `BLACKBOX`: covers each detected entity with a solid black box. */ + maskingMethod?: Skyflow.ImageV2MaskingMethod; +} diff --git a/common/src/_generated_/rest/api/types/ImageV2MaskingMethod.ts b/common/src/_generated_/rest/api/types/ImageV2MaskingMethod.ts new file mode 100644 index 00000000..fac5cb0b --- /dev/null +++ b/common/src/_generated_/rest/api/types/ImageV2MaskingMethod.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Method used to mask detected entities in the image. `BLUR`: blurs each detected entity so the sensitive content is unreadable while the rest of the image is preserved. `BLACKBOX`: covers each detected entity with a solid black box. + */ +export type ImageV2MaskingMethod = "NONE" | "BLUR" | "BLACKBOX"; +export const ImageV2MaskingMethod = { + None: "NONE", + Blur: "BLUR", + Blackbox: "BLACKBOX", +} as const; diff --git a/common/src/_generated_/rest/api/types/Media.ts b/common/src/_generated_/rest/api/types/Media.ts new file mode 100644 index 00000000..e912a812 --- /dev/null +++ b/common/src/_generated_/rest/api/types/Media.ts @@ -0,0 +1,14 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Media-specific detection and deidentification configuration. + */ +export interface Media { + audio?: Skyflow.AudioV2; + document?: Skyflow.DocumentV2; + image?: Skyflow.ImageV2; +} diff --git a/common/src/_generated_/rest/api/types/Metrics.ts b/common/src/_generated_/rest/api/types/Metrics.ts new file mode 100644 index 00000000..b791ce1e --- /dev/null +++ b/common/src/_generated_/rest/api/types/Metrics.ts @@ -0,0 +1,21 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Performance metrics for the detection and deidentification operation. + */ +export interface Metrics { + /** Size of the input text(KB). */ + size?: number; + /** Total number of words processed. */ + wordCount?: number; + /** Total number of characters processed. */ + characterCount?: number; + /** Total number of slides processed. */ + slides?: number; + /** Total number of pages processed. */ + pages?: number; + /** Total duration processed(seconds). */ + duration?: number; +} diff --git a/common/src/_generated_/rest/api/types/ProtobufAny.ts b/common/src/_generated_/rest/api/types/ProtobufAny.ts deleted file mode 100644 index 9f90a613..00000000 --- a/common/src/_generated_/rest/api/types/ProtobufAny.ts +++ /dev/null @@ -1,7 +0,0 @@ -/** - * This file was auto-generated by Fern from our API Definition. - */ - -export interface ProtobufAny { - "@type"?: string; -} diff --git a/common/src/_generated_/rest/api/types/RedactionLevel.ts b/common/src/_generated_/rest/api/types/RedactionLevel.ts new file mode 100644 index 00000000..c2a5462f --- /dev/null +++ b/common/src/_generated_/rest/api/types/RedactionLevel.ts @@ -0,0 +1,19 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Specifies a replacement pattern/type and the token group or entity type it applies to during re-identification. + */ +export interface RedactionLevel { + /** Name of the token group whose detokenized values should be masked (e.g., a token in `personTokenGroup` that detokenizes to `John Doe` is masked as `JoXXXe`). Use this for FlowVault. */ + tokenGroupName?: string; + /** Entity type whose detokenized values should be shown in plain text, redacted, or masked based on `redactionType` (e.g., a `NAME` entity that detokenizes to `John Doe` is masked as `JoXXXe`). Use this for SkyVault. */ + entityName?: Skyflow.RedactionLevelEntityName; + /** Redaction method to apply to the detokenized value (e.g., for the value `John Doe`, `MASKED` transforms it into `JoXXXe`, `REDACTED` transforms it into `*REDACTED*`, and `PLAIN_TEXT` keeps it as `John Doe`). Use this for SkyVault. */ + redactionType?: Skyflow.RedactionLevelRedactionType; + /** Redaction method to apply to the detokenized value. Use the name of a redaction pattern configured in the vault (e.g., `personMaskPattern` transforms `John Doe` into `JoXXXe`). Use this for FlowVault. */ + redactionPattern?: string; +} diff --git a/common/src/_generated_/rest/api/types/RedactionLevelEntityName.ts b/common/src/_generated_/rest/api/types/RedactionLevelEntityName.ts new file mode 100644 index 00000000..6e4e4180 --- /dev/null +++ b/common/src/_generated_/rest/api/types/RedactionLevelEntityName.ts @@ -0,0 +1,152 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Entity type whose detokenized values should be shown in plain text, redacted, or masked based on `redactionType` (e.g., a `NAME` entity that detokenizes to `John Doe` is masked as `JoXXXe`). Use this for SkyVault. + */ +export type RedactionLevelEntityName = + | "NONE" + | "BANK_ACCOUNT" + | "CREDIT_CARD" + | "CREDIT_CARD_EXPIRATION" + | "CVV" + | "DATE" + | "DATE_INTERVAL" + | "DOB" + | "DRIVER_LICENSE" + | "EMAIL_ADDRESS" + | "HEALTHCARE_NUMBER" + | "IP_ADDRESS" + | "LOCATION" + | "NAME" + | "NUMERICAL_PII" + | "PHONE_NUMBER" + | "SSN" + | "URL" + | "VEHICLE_ID" + | "MEDICAL_CODE" + | "NAME_FAMILY" + | "NAME_GIVEN" + | "ACCOUNT_NUMBER" + | "EVENT" + | "FILENAME" + | "GENDER" + | "LANGUAGE" + | "LOCATION_ADDRESS" + | "LOCATION_CITY" + | "LOCATION_COORDINATE" + | "LOCATION_COUNTRY" + | "LOCATION_STATE" + | "LOCATION_ZIP" + | "MARITAL_STATUS" + | "MONEY" + | "NAME_MEDICAL_PROFESSIONAL" + | "OCCUPATION" + | "ORGANIZATION" + | "ORGANIZATION_MEDICAL_FACILITY" + | "ORIGIN" + | "PASSPORT_NUMBER" + | "PASSWORD" + | "PHYSICAL_ATTRIBUTE" + | "POLITICAL_AFFILIATION" + | "RELIGION" + | "TIME" + | "USERNAME" + | "ZODIAC_SIGN" + | "BLOOD_TYPE" + | "CONDITION" + | "DOSE" + | "DRUG" + | "INJURY" + | "MEDICAL_PROCESS" + | "STATISTICS" + | "ROUTING_NUMBER" + | "CORPORATE_ACTION" + | "FINANCIAL_METRIC" + | "PRODUCT" + | "TREND" + | "DURATION" + | "LOCATION_ADDRESS_STREET" + | "AGE" + | "SEXUALITY" + | "EFFECT" + | "PROJECT" + | "ORGANIZATION_ID" + | "DAY" + | "MONTH" + | "YEAR" + | "ALL"; +export const RedactionLevelEntityName = { + None: "NONE", + BankAccount: "BANK_ACCOUNT", + CreditCard: "CREDIT_CARD", + CreditCardExpiration: "CREDIT_CARD_EXPIRATION", + Cvv: "CVV", + Date: "DATE", + DateInterval: "DATE_INTERVAL", + Dob: "DOB", + DriverLicense: "DRIVER_LICENSE", + EmailAddress: "EMAIL_ADDRESS", + HealthcareNumber: "HEALTHCARE_NUMBER", + IpAddress: "IP_ADDRESS", + Location: "LOCATION", + Name: "NAME", + NumericalPii: "NUMERICAL_PII", + PhoneNumber: "PHONE_NUMBER", + Ssn: "SSN", + Url: "URL", + VehicleId: "VEHICLE_ID", + MedicalCode: "MEDICAL_CODE", + NameFamily: "NAME_FAMILY", + NameGiven: "NAME_GIVEN", + AccountNumber: "ACCOUNT_NUMBER", + Event: "EVENT", + Filename: "FILENAME", + Gender: "GENDER", + Language: "LANGUAGE", + LocationAddress: "LOCATION_ADDRESS", + LocationCity: "LOCATION_CITY", + LocationCoordinate: "LOCATION_COORDINATE", + LocationCountry: "LOCATION_COUNTRY", + LocationState: "LOCATION_STATE", + LocationZip: "LOCATION_ZIP", + MaritalStatus: "MARITAL_STATUS", + Money: "MONEY", + NameMedicalProfessional: "NAME_MEDICAL_PROFESSIONAL", + Occupation: "OCCUPATION", + Organization: "ORGANIZATION", + OrganizationMedicalFacility: "ORGANIZATION_MEDICAL_FACILITY", + Origin: "ORIGIN", + PassportNumber: "PASSPORT_NUMBER", + Password: "PASSWORD", + PhysicalAttribute: "PHYSICAL_ATTRIBUTE", + PoliticalAffiliation: "POLITICAL_AFFILIATION", + Religion: "RELIGION", + Time: "TIME", + Username: "USERNAME", + ZodiacSign: "ZODIAC_SIGN", + BloodType: "BLOOD_TYPE", + Condition: "CONDITION", + Dose: "DOSE", + Drug: "DRUG", + Injury: "INJURY", + MedicalProcess: "MEDICAL_PROCESS", + Statistics: "STATISTICS", + RoutingNumber: "ROUTING_NUMBER", + CorporateAction: "CORPORATE_ACTION", + FinancialMetric: "FINANCIAL_METRIC", + Product: "PRODUCT", + Trend: "TREND", + Duration: "DURATION", + LocationAddressStreet: "LOCATION_ADDRESS_STREET", + Age: "AGE", + Sexuality: "SEXUALITY", + Effect: "EFFECT", + Project: "PROJECT", + OrganizationId: "ORGANIZATION_ID", + Day: "DAY", + Month: "MONTH", + Year: "YEAR", + All: "ALL", +} as const; diff --git a/common/src/_generated_/rest/api/types/RedactionLevelRedactionType.ts b/common/src/_generated_/rest/api/types/RedactionLevelRedactionType.ts new file mode 100644 index 00000000..d3258424 --- /dev/null +++ b/common/src/_generated_/rest/api/types/RedactionLevelRedactionType.ts @@ -0,0 +1,14 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Redaction method to apply to the detokenized value (e.g., for the value `John Doe`, `MASKED` transforms it into `JoXXXe`, `REDACTED` transforms it into `*REDACTED*`, and `PLAIN_TEXT` keeps it as `John Doe`). Use this for SkyVault. + */ +export type RedactionLevelRedactionType = "DEFAULT" | "REDACTED" | "MASKED" | "PLAIN_TEXT"; +export const RedactionLevelRedactionType = { + Default: "DEFAULT", + Redacted: "REDACTED", + Masked: "MASKED", + PlainText: "PLAIN_TEXT", +} as const; diff --git a/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2.ts b/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2.ts new file mode 100644 index 00000000..b1739ed9 --- /dev/null +++ b/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2.ts @@ -0,0 +1,18 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Response containing the status and output of a file reidentification job. + */ +export interface ReidentifyFileResponseV2 { + /** Status of the reidentification request. */ + status?: Skyflow.ReidentifyFileResponseV2Status; + /** Data source of the output file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. */ + outputType?: Skyflow.ReidentifyFileResponseV2OutputType; + /** Details of output files */ + output?: Skyflow.FileOutput[]; + metrics?: Skyflow.Metrics; +} diff --git a/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2OutputType.ts b/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2OutputType.ts new file mode 100644 index 00000000..7cf99e60 --- /dev/null +++ b/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2OutputType.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Data source of the output file. `BASE64`: Base64-encoded file string. `SKYFLOW_ID`: Reference file by vault ID. `PRESIGNED_URL`: S3 presigned URL of input file. + */ +export type ReidentifyFileResponseV2OutputType = "BASE64" | "SKYFLOW_ID" | "PRESIGNED_URL"; +export const ReidentifyFileResponseV2OutputType = { + Base64: "BASE64", + SkyflowId: "SKYFLOW_ID", + PresignedUrl: "PRESIGNED_URL", +} as const; diff --git a/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2Status.ts b/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2Status.ts new file mode 100644 index 00000000..99b0ff04 --- /dev/null +++ b/common/src/_generated_/rest/api/types/ReidentifyFileResponseV2Status.ts @@ -0,0 +1,15 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Status of the reidentification request. + */ +export type ReidentifyFileResponseV2Status = "UNKNOWN" | "FAILED" | "SUCCESS" | "IN_PROGRESS" | "QUEUED"; +export const ReidentifyFileResponseV2Status = { + Unknown: "UNKNOWN", + Failed: "FAILED", + Success: "SUCCESS", + InProgress: "IN_PROGRESS", + Queued: "QUEUED", +} as const; diff --git a/common/src/_generated_/rest/api/types/ReidentifyStringResponseV2.ts b/common/src/_generated_/rest/api/types/ReidentifyStringResponseV2.ts new file mode 100644 index 00000000..25123771 --- /dev/null +++ b/common/src/_generated_/rest/api/types/ReidentifyStringResponseV2.ts @@ -0,0 +1,14 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +import * as Skyflow from "../index"; + +/** + * Response containing the reidentified string with tokens replaced by original values. + */ +export interface ReidentifyStringResponseV2 { + /** Reidentified text. */ + processedText?: string; + metrics?: Skyflow.Metrics; +} diff --git a/common/src/_generated_/rest/api/types/TransformationShiftDates.ts b/common/src/_generated_/rest/api/types/TransformationShiftDates.ts new file mode 100644 index 00000000..f16bf304 --- /dev/null +++ b/common/src/_generated_/rest/api/types/TransformationShiftDates.ts @@ -0,0 +1,13 @@ +/** + * This file was auto-generated by Fern from our API Definition. + */ + +/** + * Shifts detected date values by a random number of days within the given range. Negative values shift dates backward and positive values shift them forward. + */ +export interface TransformationShiftDates { + /** Minimum number of days to shift dates. */ + minDays?: number; + /** Maximum number of days to shift dates. */ + maxDays?: number; +} diff --git a/common/src/_generated_/rest/api/types/index.ts b/common/src/_generated_/rest/api/types/index.ts index 63cf5e7a..228fc7fe 100644 --- a/common/src/_generated_/rest/api/types/index.ts +++ b/common/src/_generated_/rest/api/types/index.ts @@ -1,3 +1,51 @@ -export * from "./V1GetAuthTokenResponse"; -export * from "./GooglerpcStatus"; -export * from "./ProtobufAny"; +export * from "./GetAuthTokenResponse"; +export * from "./GetStsTokenResponse"; +export * from "./HttpCode"; +export * from "./ErrorResponseError"; +export * from "./ErrorResponse"; +export * from "./AudioV2OutputTranscription"; +export * from "./AudioV2"; +export * from "./AudioV2Bleep"; +export * from "./DeidentifyFileResponseV2"; +export * from "./DeidentifyStringResponseV2"; +export * from "./DeidentifyStringResponseV2DetectedEntity"; +export * from "./DeidentifyStringResponseV2EntityLocation"; +export * from "./DetectReturnEntities"; +export * from "./Detect"; +export * from "./DetectConfigV2VaultType"; +export * from "./DetectConfigV2"; +export * from "./DetectGuardrailsResponseV2Validation"; +export * from "./DetectGuardrailsResponseV2"; +export * from "./DetectRunsResponseV2Status"; +export * from "./DetectRunsResponseV2OutputType"; +export * from "./DetectRunsResponseV2"; +export * from "./DetectCustomTypesDeidentificationType"; +export * from "./DetectCustomTypesIncludePrefix"; +export * from "./DetectCustomTypes"; +export * from "./DetectEntityEntityType"; +export * from "./DetectEntityDeidentificationType"; +export * from "./DetectEntityIncludePrefix"; +export * from "./DetectEntity"; +export * from "./DetectObjectEntityEntityType"; +export * from "./DetectObjectEntityDeidentificationType"; +export * from "./DetectObjectEntity"; +export * from "./DocumentV2"; +export * from "./DocumentV2PdfProcessingMode"; +export * from "./DocumentV2Pdf"; +export * from "./EntityTransformation"; +export * from "./FileMapping"; +export * from "./FileOutputProcessedFileType"; +export * from "./FileOutputProcessedFileExtension"; +export * from "./FileOutput"; +export * from "./ImageV2MaskingMethod"; +export * from "./ImageV2"; +export * from "./Media"; +export * from "./Metrics"; +export * from "./RedactionLevelEntityName"; +export * from "./RedactionLevelRedactionType"; +export * from "./RedactionLevel"; +export * from "./ReidentifyFileResponseV2Status"; +export * from "./ReidentifyFileResponseV2OutputType"; +export * from "./ReidentifyFileResponseV2"; +export * from "./ReidentifyStringResponseV2"; +export * from "./TransformationShiftDates"; diff --git a/common/src/_generated_/rest/environments.ts b/common/src/_generated_/rest/environments.ts index 4a1edc91..c5f61083 100644 --- a/common/src/_generated_/rest/environments.ts +++ b/common/src/_generated_/rest/environments.ts @@ -3,8 +3,8 @@ */ export const SkyflowEnvironment = { - Production: "https://manage.skyflowapis.com", - Sandbox: "https://manage.skyflowapis-preview.com", + Production: "https://%7B%7Bvault_url%7D%7D", + Sandbox: "https://%7B%7Bvault_url%7D%7D", } as const; export type SkyflowEnvironment = typeof SkyflowEnvironment.Production | typeof SkyflowEnvironment.Sandbox; diff --git a/common/src/detect/controller/index.ts b/common/src/detect/controller/index.ts new file mode 100644 index 00000000..cc966dfe --- /dev/null +++ b/common/src/detect/controller/index.ts @@ -0,0 +1,333 @@ +// ============================================================================= +// BaseDetectController — the shared implementation of the Skyflow Detect V2 API, +// analogous to Java's `common` BaseDetectController. Each SDK's DetectController +// extends it and supplies only its own VaultClient, which already knows the +// vault URL, the vault id, the log level, how to resolve a bearer token, and any +// per-vault timeout / retry overrides. +// ============================================================================= +import { StringsV2 } from "../../_generated_/rest/api/resources/stringsV2/client/Client"; +import { FilesV2 } from "../../_generated_/rest/api/resources/filesV2/client/Client"; +import { GuardrailsV2 } from "../../_generated_/rest/api/resources/guardrailsV2/client/Client"; +import { GetRunV2Request } from "../../_generated_/rest/api"; +import { LogLevel, MessageType, SDK, generateSDKMetrics, parameterizedString, printLog } from "../../utils"; +import logs from "../../utils/logs"; +import DeidentifyStringRequest from "../model/request/deidentify-string"; +import DeidentifyFileRequest from "../model/request/deidentify-file"; +import ReidentifyStringRequest from "../model/request/reidentify-string"; +import ReidentifyFileRequest from "../model/request/reidentify-file"; +import GetRunRequest from "../model/request/get-run"; +import CheckGuardrailsRequest from "../model/request/check-guardrails"; +import DeidentifyStringResponse from "../model/response/deidentify-string"; +import DeidentifyFileResponse from "../model/response/deidentify-file"; +import ReidentifyStringResponse from "../model/response/reidentify-string"; +import ReidentifyFileResponse from "../model/response/reidentify-file"; +import GetRunResponse from "../model/response/get-run"; +import CheckGuardrailsResponse from "../model/response/check-guardrails"; +import { + validateCheckGuardrailsRequest, + validateDeidentifyFileRequest, + validateDeidentifyStringRequest, + validateGetRunRequest, + validateReidentifyFileRequest, + validateReidentifyStringRequest, +} from "../internal/validations"; +import { + toCheckGuardrailsRequest, + toDeidentifyFileRequest, + toDeidentifyStringRequest, + toReidentifyFileRequest, + toReidentifyStringRequest, +} from "../internal/request-mapper"; +import { + enumOrDefault, + toCheckGuardrailsResponse, + toDeidentifyStringResponse, + toGetRunResponse, + toReidentifyFileResponse, + toReidentifyStringResponse, +} from "../internal/response-mapper"; +import { DeidentifyFileSource, ReidentifyFileSource } from "../types"; +import { completeDeidentifyFile } from "../internal/deidentify-file-flow"; +import { toSkyflowError } from "../internal/errors"; + +/** The slice of a layer's VaultClient the shared Detect controller depends on. */ +export interface DetectClientLike { + /** The vault every request is scoped to. */ + readonly vaultId: string; + /** The vault base URL requests are sent to, for example `https://.vault.skyflowapis.com`. */ + readonly url: string; + getLogLevel(): LogLevel; + /** Resolves the bearer token or API key for the next call, refreshing it when expired. */ + resolveAuthToken(): Promise; + /** Overall per-call timeout in seconds; undefined for the transport default. */ + readonly timeout?: number; + /** Retries on retryable HTTP statuses; undefined for the transport default. */ + readonly maxRetries?: number; +} + +/** Per-call options accepted by every generated Detect V2 resource client. */ +type DetectRequestOptions = StringsV2.RequestOptions; + +export abstract class BaseDetectController { + protected readonly client: TClient; + + private currentToken = ""; + + private stringsApi?: StringsV2; + + private filesApi?: FilesV2; + + private guardrailsApi?: GuardrailsV2; + + constructor(client: TClient) { + this.client = client; + printLog( + parameterizedString(logs.infoLogs.DETECT_CONTROLLER_INITIALIZED, client.vaultId), + MessageType.LOG, + client.getLogLevel(), + ); + } + + protected get logLevel(): LogLevel { + return this.client.getLogLevel(); + } + + // ------------------------------------------------------------------ + // Vault-specific types + // ------------------------------------------------------------------ + + /** + * Resolves the `outputType` of a de-identification run onto this SDK's `DeidentifySourceType`. + * Returns `null` for a wire value that enum does not contain, so a vault never surfaces a source + * the SDK does not offer. + */ + protected abstract toDeidentifySource(value: string | null | undefined): DeidentifyFileSource | null; + + /** Resolves the `outputType` of a re-identification onto this SDK's `ReidentifySourceType`; `null` when unknown. */ + protected abstract toReidentifySource(value: string | null | undefined): ReidentifyFileSource | null; + + /** Maps a wire value onto one of `values`' members, or `null` when it is not one of them. */ + protected static enumOrNull(values: Record, value: string | null | undefined): E | null { + return enumOrDefault(values, value, null); + } + + // ------------------------------------------------------------------ + // Detect V2 operations + // ------------------------------------------------------------------ + + /** + * Detects and de-identifies sensitive data in a string. Synchronous. + * + * @param request text plus, optionally, a stored configuration id or an inline configuration + * @returns processed text, detected entities and metrics + * @throws SkyflowError on validation failure, credential failure or an API error + */ + async deidentifyString(request: DeidentifyStringRequest): Promise { + printLog(logs.infoLogs.DEIDENTIFY_STRING_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_DEIDENTIFY_STRING_REQUEST, MessageType.LOG, this.logLevel); + validateDeidentifyStringRequest(request, this.logLevel); + const options = await this.prepareCall(); + const apiRequest = toDeidentifyStringRequest(request, this.client.vaultId); + try { + const apiResponse = await this.strings().deidentifyStringV2(apiRequest, options); + printLog(logs.infoLogs.DEIDENTIFY_STRING_REQUEST_RESOLVED, MessageType.LOG, this.logLevel); + const response = toDeidentifyStringResponse(apiResponse); + printLog(logs.infoLogs.DEIDENTIFY_STRING_SUCCESS, MessageType.LOG, this.logLevel); + return response; + } catch (err) { + throw toSkyflowError(err, logs.errorLogs.DEIDENTIFY_STRING_REQUEST_REJECTED, this.logLevel); + } + } + + /** + * Replaces tokens in a string with their original values or a redacted rendering. Synchronous. + * + * @param request text containing tokens plus optional per-group redaction rules + * @returns re-identified text and metrics + * @throws SkyflowError on validation failure, credential failure or an API error + */ + async reidentifyString(request: ReidentifyStringRequest): Promise { + printLog(logs.infoLogs.REIDENTIFY_STRING_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_REIDENTIFY_STRING_REQUEST, MessageType.LOG, this.logLevel); + validateReidentifyStringRequest(request, this.logLevel); + const options = await this.prepareCall(); + const apiRequest = toReidentifyStringRequest(request, this.client.vaultId); + try { + const apiResponse = await this.strings().reidentifyStringV2(apiRequest, options); + printLog(logs.infoLogs.REIDENTIFY_STRING_REQUEST_RESOLVED, MessageType.LOG, this.logLevel); + const response = toReidentifyStringResponse(apiResponse); + printLog(logs.infoLogs.REIDENTIFY_STRING_SUCCESS, MessageType.LOG, this.logLevel); + return response; + } catch (err) { + throw toSkyflowError(err, logs.errorLogs.REIDENTIFY_STRING_REQUEST_REJECTED, this.logLevel); + } + } + + /** + * Detects and de-identifies sensitive data in a file. The API is asynchronous: by default this + * submits the run and returns its id for use with {@link getRun}. With `pollOptions` the SDK + * polls the run with exponential backoff and returns the completed run, or the run id with a + * non-terminal status if the budget runs out. On success, a request built from a local file + * (`file` or `filePath`) has its outputs written to disk. + * + * @param request the file (base64, skyflow id, presigned URL, or a local file) plus a stored or inline configuration + * @returns run id, and the run's status, outputs and metrics when polled + * @throws SkyflowError on validation failure, credential failure, an API error or an output write failure + */ + async deidentifyFile(request: DeidentifyFileRequest): Promise { + printLog(logs.infoLogs.DEIDENTIFY_FILE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_DEIDENTIFY_FILE_REQUEST, MessageType.LOG, this.logLevel); + validateDeidentifyFileRequest(request, this.logLevel); + const options = await this.prepareCall(); + const apiRequest = await toDeidentifyFileRequest(request, this.client.vaultId, this.logLevel); + let runId: string | null; + try { + const apiResponse = await this.files().deidentifyFileV2(apiRequest, options); + printLog(logs.infoLogs.DEIDENTIFY_FILE_REQUEST_RESOLVED, MessageType.LOG, this.logLevel); + runId = apiResponse.runId ?? null; + } catch (err) { + throw toSkyflowError(err, logs.errorLogs.DEIDENTIFY_FILE_REQUEST_REJECTED, this.logLevel); + } + const response = await completeDeidentifyFile( + request, + runId, + (id) => this.getRun(new GetRunRequest(id)), + this.logLevel, + this.sleeper(), + ); + printLog(logs.infoLogs.DEIDENTIFY_FILE_SUCCESS, MessageType.LOG, this.logLevel); + return response; + } + + /** + * Replaces tokens in a file with their original values or a redacted rendering. Synchronous: + * the re-identified file is returned in the response rather than through a run id. + * + * @param request the file (base64 or presigned URL) plus optional per-group redaction rules + * @returns status, output files and metrics + * @throws SkyflowError on validation failure, credential failure or an API error + */ + async reidentifyFile(request: ReidentifyFileRequest): Promise { + printLog(logs.infoLogs.REIDENTIFY_FILE_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_REIDENTIFY_FILE_REQUEST, MessageType.LOG, this.logLevel); + validateReidentifyFileRequest(request, this.logLevel); + const options = await this.prepareCall(); + const apiRequest = toReidentifyFileRequest(request, this.client.vaultId); + try { + const apiResponse = await this.files().reidentifyFileV2(apiRequest, options); + printLog(logs.infoLogs.REIDENTIFY_FILE_REQUEST_RESOLVED, MessageType.LOG, this.logLevel); + const response = toReidentifyFileResponse(apiResponse, (value) => this.toReidentifySource(value)); + printLog(logs.infoLogs.REIDENTIFY_FILE_SUCCESS, MessageType.LOG, this.logLevel); + return response; + } catch (err) { + throw toSkyflowError(err, logs.errorLogs.REIDENTIFY_FILE_REQUEST_REJECTED, this.logLevel); + } + } + + /** + * Retrieves the status and, once complete, the outputs of a Detect file run. + * + * @param request the run id returned by `deidentifyFile` + * @returns status, output files, metrics and failure message + * @throws SkyflowError on validation failure, credential failure or an API error + */ + async getRun(request: GetRunRequest): Promise { + printLog(logs.infoLogs.GET_RUN_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_GET_RUN_REQUEST, MessageType.LOG, this.logLevel); + validateGetRunRequest(request, this.logLevel); + const options = await this.prepareCall(); + const runId = request.runId.trim(); + const apiRequest: GetRunV2Request = { vaultId: this.client.vaultId }; + try { + const apiResponse = await this.files().getRunV2(runId, apiRequest, options); + printLog(logs.infoLogs.GET_RUN_REQUEST_RESOLVED, MessageType.LOG, this.logLevel); + const response = toGetRunResponse(runId, apiResponse, (value) => this.toDeidentifySource(value)); + printLog(logs.infoLogs.GET_RUN_SUCCESS, MessageType.LOG, this.logLevel); + return response; + } catch (err) { + throw toSkyflowError(err, logs.errorLogs.GET_RUN_REQUEST_REJECTED, this.logLevel); + } + } + + /** + * Screens text for toxicity and denied topics. Synchronous. + * + * @param request text plus the checks to apply + * @returns the overall validation result and the individual check outcomes + * @throws SkyflowError on validation failure, credential failure or an API error + */ + async checkGuardrails(request: CheckGuardrailsRequest): Promise { + printLog(logs.infoLogs.CHECK_GUARDRAILS_TRIGGERED, MessageType.LOG, this.logLevel); + printLog(logs.infoLogs.VALIDATE_CHECK_GUARDRAILS_REQUEST, MessageType.LOG, this.logLevel); + validateCheckGuardrailsRequest(request, this.logLevel); + const options = await this.prepareCall(); + const apiRequest = toCheckGuardrailsRequest(request, this.client.vaultId); + try { + const apiResponse = await this.guardrails().checkGuardrailsV2(apiRequest, options); + printLog(logs.infoLogs.CHECK_GUARDRAILS_REQUEST_RESOLVED, MessageType.LOG, this.logLevel); + const response = toCheckGuardrailsResponse(apiResponse); + printLog(logs.infoLogs.CHECK_GUARDRAILS_SUCCESS, MessageType.LOG, this.logLevel); + return response; + } catch (err) { + throw toSkyflowError(err, logs.errorLogs.CHECK_GUARDRAILS_REQUEST_REJECTED, this.logLevel); + } + } + + // ------------------------------------------------------------------ + // Transport plumbing + // ------------------------------------------------------------------ + + /** Resolves the token for the next call and builds its per-call options. */ + private async prepareCall(): Promise { + this.currentToken = await this.client.resolveAuthToken(); + const options: DetectRequestOptions = { + headers: { [SDK.METRICS_HEADER_KEY]: JSON.stringify(generateSDKMetrics(this.logLevel)) }, + }; + if (this.client.timeout !== undefined) { + options.timeoutInSeconds = this.client.timeout; + } + if (this.client.maxRetries !== undefined) { + options.maxRetries = this.client.maxRetries; + } + return options; + } + + /** Suppliers read the URL and token live, so a config update or token refresh needs no rebuild. */ + private supplierOptions(): StringsV2.Options { + return { + baseUrl: () => this.client.url, + token: () => this.currentToken, + }; + } + + /** The generated strings client, built on first use. */ + protected strings(): StringsV2 { + if (!this.stringsApi) { + this.stringsApi = new StringsV2(this.supplierOptions()); + } + return this.stringsApi; + } + + /** The generated files client, built on first use. */ + protected files(): FilesV2 { + if (!this.filesApi) { + this.filesApi = new FilesV2(this.supplierOptions()); + } + return this.filesApi; + } + + /** The generated guardrails client, built on first use. */ + protected guardrails(): GuardrailsV2 { + if (!this.guardrailsApi) { + this.guardrailsApi = new GuardrailsV2(this.supplierOptions()); + } + return this.guardrailsApi; + } + + /** The pause used between run lookups while polling; undefined selects the real timer. Test seam. */ + protected sleeper(): ((millis: number) => Promise) | undefined { + return undefined; + } +} + +export default BaseDetectController; diff --git a/common/src/detect/index.ts b/common/src/detect/index.ts new file mode 100644 index 00000000..c4fd3ae9 --- /dev/null +++ b/common/src/detect/index.ts @@ -0,0 +1,27 @@ +// ============================================================================= +// Public Detect V2 surface shared by the skyvault and flowvault SDKs. +// ============================================================================= +export * from "./types"; + +export { default as DeidentifyStringRequest } from "./model/request/deidentify-string"; +export type { DeidentifyStringRequestFields } from "./model/request/deidentify-string"; +export { default as DeidentifyFileRequest } from "./model/request/deidentify-file"; +export type { DeidentifyFileRequestFields } from "./model/request/deidentify-file"; +export { default as ReidentifyStringRequest } from "./model/request/reidentify-string"; +export type { ReidentifyStringRequestFields } from "./model/request/reidentify-string"; +export { default as ReidentifyFileRequest } from "./model/request/reidentify-file"; +export type { ReidentifyFileRequestFields } from "./model/request/reidentify-file"; +export { default as GetRunRequest } from "./model/request/get-run"; +export { default as CheckGuardrailsRequest } from "./model/request/check-guardrails"; +export type { CheckGuardrailsRequestFields } from "./model/request/check-guardrails"; + +export { default as DeidentifyStringResponse } from "./model/response/deidentify-string"; +export { default as DeidentifyFileResponse } from "./model/response/deidentify-file"; +export { default as ReidentifyStringResponse } from "./model/response/reidentify-string"; +export { default as ReidentifyFileResponse } from "./model/response/reidentify-file"; +export { default as GetRunResponse } from "./model/response/get-run"; +export type { DetectRunFields } from "./model/response/get-run"; +export { default as CheckGuardrailsResponse } from "./model/response/check-guardrails"; + +export { BaseDetectController } from "./controller"; +export type { DetectClientLike } from "./controller"; diff --git a/common/src/detect/internal/deidentify-file-flow.ts b/common/src/detect/internal/deidentify-file-flow.ts new file mode 100644 index 00000000..47646904 --- /dev/null +++ b/common/src/detect/internal/deidentify-file-flow.ts @@ -0,0 +1,37 @@ +// Everything `deidentifyFile` does after the submit call, shared by every SDK: decide whether to +// poll, poll, merge the run into the response, and write outputs for local-file requests. Each +// controller performs the submit itself and hands the run id here. +import { LogLevel, MessageType, parameterizedString, printLog } from "../../utils"; +import logs from "../../utils/logs"; +import { DetectRunStatus } from "../types"; +import DeidentifyFileRequest from "../model/request/deidentify-file"; +import DeidentifyFileResponse from "../model/response/deidentify-file"; +import { localFileName } from "./request-mapper"; +import { submittedDeidentifyFileResponse, toDeidentifyFileResponse } from "./response-mapper"; +import { pollRun, RunFetcher, Sleeper } from "./run-poller"; +import { writeOutputs } from "./file-output-writer"; + +/** @internal */ +export async function completeDeidentifyFile( + request: DeidentifyFileRequest, + runId: string | null, + fetcher: RunFetcher, + logLevel: LogLevel = LogLevel.ERROR, + sleeper?: Sleeper, +): Promise { + if (!request.pollOptions || !runId) { + printLog( + parameterizedString(logs.infoLogs.DEIDENTIFY_FILE_RUN_SUBMITTED, String(runId)), + MessageType.LOG, + logLevel, + ); + return submittedDeidentifyFileResponse(runId); + } + const run = await pollRun(runId, request.pollOptions, fetcher, logLevel, sleeper); + const response = toDeidentifyFileResponse(runId, run); + const inputName = localFileName(request); + if (response.status === DetectRunStatus.SUCCESS && inputName) { + await writeOutputs(response.output, response.outputType, inputName, request.outputDirectory, logLevel); + } + return response; +} diff --git a/common/src/detect/internal/errors.ts b/common/src/detect/internal/errors.ts new file mode 100644 index 00000000..8209563d --- /dev/null +++ b/common/src/detect/internal/errors.ts @@ -0,0 +1,43 @@ +// Normalizes a failure from the generated Detect V2 client into the SDK's SkyflowError. +// The wire envelope is `{ error: { grpc_code, http_code, http_status, message, details } }`, +// reached through the generated error's `body`; the request id travels in the `x-request-id` +// response header. +import { HTTP_HEADER, HTTP_STATUS_CODE, LogLevel, MessageType, printLog } from "../../utils"; +import errorMessages from "../../error/messages"; +import SkyflowError from "../../error"; + +interface GeneratedErrorLike { + statusCode?: number; + body?: unknown; + rawResponse?: { headers?: { get?: (name: string) => string | null | undefined } }; + message?: string; +} + +/** @internal */ +export function toSkyflowError(err: unknown, errorLog: string, logLevel: LogLevel = LogLevel.ERROR): SkyflowError { + printLog(errorLog, MessageType.ERROR, logLevel); + if (err instanceof SkyflowError) { + return err; + } + const e = (err ?? {}) as GeneratedErrorLike; + const headers = e.rawResponse?.headers; + const headerRequestId = headers && headers.get ? headers.get(HTTP_HEADER.X_REQUEST_ID) ?? "" : ""; + + const body = e.body; + const errorData = + ((body && typeof body === "object" ? (body as { error?: unknown }).error : undefined) ?? {}) as Record; + + const message = String(errorData.message ?? e.message ?? errorMessages.ERROR_OCCURRED); + const httpCode = + (e.statusCode as number | undefined) ?? + (errorData.httpCode as number | string | undefined) ?? + (errorData.http_code as number | string | undefined) ?? + HTTP_STATUS_CODE.BAD_REQUEST; + const grpcCode = (errorData.grpcCode ?? errorData.grpc_code) as number | string | undefined; + const httpStatus = (errorData.httpStatus ?? errorData.http_status) as string | undefined; + const details = (errorData.details ?? []) as Array; + const requestId = (errorData.requestId as string | undefined) ?? headerRequestId; + + printLog(message, MessageType.ERROR, logLevel); + return new SkyflowError({ httpCode, message, requestId, grpcCode, httpStatus, details }, []); +} diff --git a/common/src/detect/internal/file-output-writer.ts b/common/src/detect/internal/file-output-writer.ts new file mode 100644 index 00000000..c3ff8e49 --- /dev/null +++ b/common/src/detect/internal/file-output-writer.ts @@ -0,0 +1,114 @@ +// Writes the base64 outputs of a successful file run next to, or into a chosen directory for, the +// local file that was submitted. Internal to the SDK. +// +// Each output becomes `processed-.`. Entity outputs carry their type in the +// name so they never collide with a processed file of the same extension: ENTITIES becomes +// `processed-entities-.json` and OBJECT_ENTITIES becomes +// `processed-object-entities-.json`. If two outputs would still share a name the file +// type is added before the extension. Presigned-URL outputs are not fetched and nothing is written +// for them. +import * as fs from "fs"; +import * as path from "path"; +import { ENCODING_TYPE, LogLevel, MessageType, parameterizedString, printLog } from "../../utils"; +import logs from "../../utils/logs"; +import SkyflowError from "../../error"; +import SKYFLOW_ERROR_CODE from "../../error/codes"; +import { BASE64_SOURCE, DeidentifyFileSource, FileOutput } from "../types"; + +/** @internal */ +export const PROCESSED_FILE_NAME_PREFIX = "processed-"; + +const ENTITIES_TYPE = "ENTITIES"; + +/** @internal */ +export function stripExtension(fileName: string | null | undefined): string { + if (!fileName) { + return "output"; + } + const ext = path.extname(fileName); + return ext && ext.length < fileName.length ? fileName.slice(0, -ext.length) : fileName; +} + +/** + * @param base the input file name without its extension + * @returns `processed-.`, or `processed--.json` for entity outputs + * @internal + */ +export function outputFileName(base: string, output: FileOutput, used: Set): string { + const type = output.processedFileType ?? ""; + const entities = type.toUpperCase().includes(ENTITIES_TYPE); + let extension: string; + if (output.processedFileExtension) { + extension = output.processedFileExtension.toLowerCase(); + } else if (entities) { + extension = "json"; + } else { + extension = "bin"; + } + // ENTITIES -> processed-entities-, OBJECT_ENTITIES -> processed-object-entities- + const stem = entities + ? `${PROCESSED_FILE_NAME_PREFIX}${type.toLowerCase().replace(/_/g, "-")}-${base}` + : `${PROCESSED_FILE_NAME_PREFIX}${base}`; + let name = `${stem}.${extension}`; + if (used.has(name)) { + name = `${stem}.${type.length === 0 ? String(used.size) : type.toLowerCase()}.${extension}`; + } + used.add(name); + return name; +} + +function writeFailure(log: string, target: string, logLevel: LogLevel): SkyflowError { + printLog(parameterizedString(log, target), MessageType.ERROR, logLevel); + return new SkyflowError(SKYFLOW_ERROR_CODE.FAILED_TO_WRITE_OUTPUT_FILE, [target]); +} + +/** + * @returns the paths written, in output order; empty when the output type is not BASE64 + * @throws SkyflowError if the directory cannot be created or a file cannot be decoded or written + * @internal + */ +export async function writeOutputs( + outputs: FileOutput[], + outputType: DeidentifyFileSource | null, + inputFileName: string, + outputDirectory: string | undefined, + logLevel: LogLevel = LogLevel.ERROR, +): Promise { + const written: string[] = []; + if (!outputs || outputs.length === 0) { + return written; + } + if (outputType !== BASE64_SOURCE) { + printLog( + parameterizedString(logs.infoLogs.OUTPUT_NOT_WRITTEN_FOR_OUTPUT_TYPE, String(outputType)), + MessageType.LOG, + logLevel, + ); + return written; + } + const directory = outputDirectory && outputDirectory.trim().length > 0 ? outputDirectory : undefined; + if (directory) { + try { + await fs.promises.mkdir(directory, { recursive: true }); + } catch (_e) { + throw writeFailure(logs.errorLogs.OUTPUT_DIRECTORY_NOT_CREATED, directory, logLevel); + } + } + const base = stripExtension(inputFileName); + const used = new Set(); + for (const output of outputs) { + if (!output || !output.processedFile) { + continue; + } + const name = outputFileName(base, output, used); + const target = directory ? path.join(directory, name) : name; + try { + await fs.promises.writeFile(target, Buffer.from(output.processedFile, ENCODING_TYPE.BASE64)); + } catch (_e) { + throw writeFailure(logs.errorLogs.OUTPUT_FILE_NOT_WRITTEN, target, logLevel); + } + printLog(parameterizedString(logs.infoLogs.OUTPUT_FILE_WRITTEN, target), MessageType.LOG, logLevel); + written.push(target); + } + return written; +} diff --git a/common/src/detect/internal/request-mapper.ts b/common/src/detect/internal/request-mapper.ts new file mode 100644 index 00000000..502bad93 --- /dev/null +++ b/common/src/detect/internal/request-mapper.ts @@ -0,0 +1,339 @@ +// Converts public SDK request models into generated V2 API requests. Internal to the SDK. +import * as fs from "fs"; +import * as path from "path"; +import { + AudioV2, + DeidentifyFileRequestV2, + DeidentifyFileRequestV2DataFormat, + DeidentifyFileRequestV2DataSource, + DeidentifyStringRequestV2, + Detect as DetectV2, + DetectConfigV2, + DetectCustomTypes, + DetectEntity, + DetectGuardrailsRequestV2, + DetectObjectEntity, + DocumentV2, + EntityTransformation, + FileMapping as FileMappingV2, + ImageV2, + Media as MediaV2, + RedactionLevel as RedactionLevelV2, + ReidentifyFileRequestV2, + ReidentifyFileRequestV2DataFormat, + ReidentifyFileRequestV2DataSource, + ReidentifyStringRequestV2, +} from "../../_generated_/rest/api"; +import { ENCODING_TYPE, LogLevel, MessageType, parameterizedString, printLog } from "../../utils"; +import logs from "../../utils/logs"; +import SkyflowError from "../../error"; +import SKYFLOW_ERROR_CODE from "../../error/codes"; +import { + Audio, + BASE64_SOURCE, + CustomType, + Detect, + DetectConfiguration, + Document, + Entity, + FileDataFormat, + FileMapping, + Image, + Media, + ObjectEntity, + RedactionRule, + Transformation, +} from "../types"; +import DeidentifyStringRequest from "../model/request/deidentify-string"; +import DeidentifyFileRequest from "../model/request/deidentify-file"; +import ReidentifyStringRequest from "../model/request/reidentify-string"; +import ReidentifyFileRequest from "../model/request/reidentify-file"; +import CheckGuardrailsRequest from "../model/request/check-guardrails"; +import { DETECT_INTERFACE_NAME } from "./validations"; + +/** + * The API needs either a configuration id or a configuration, and the vault id travels inside the + * configuration. An inline configuration is always sent; with neither field set the SDK still sends a + * configuration holding only the vault id, so the server applies the vault's defaults instead of + * rejecting the request. When only an id is set nothing else is sent. + * @internal + */ +export function sendsConfiguration(configurationId?: string, configuration?: DetectConfiguration): boolean { + return configuration !== undefined || configurationId === undefined; +} + +/** @internal */ +export function toDeidentifyStringRequest(request: DeidentifyStringRequest, vaultId: string): DeidentifyStringRequestV2 { + const apiRequest: DeidentifyStringRequestV2 = { text: request.text }; + if (request.configurationId !== undefined) { + apiRequest.configurationId = request.configurationId; + } + if (sendsConfiguration(request.configurationId, request.configuration)) { + apiRequest.configuration = toDetectConfig(request.configuration, vaultId); + } + return apiRequest; +} + +/** The name of the local file a request was built from, or null for value-based requests. @internal */ +export function localFileName(request: DeidentifyFileRequest): string | null { + if (request.file) { + const name = (request.file as { name?: unknown }).name; + return typeof name === "string" && name.length > 0 ? path.basename(name) : "output"; + } + if (request.filePath) { + return path.basename(request.filePath); + } + return null; +} + +async function encodeLocalFile(request: DeidentifyFileRequest, logLevel: LogLevel): Promise { + try { + if (request.file) { + const bytes = await request.file.arrayBuffer(); + return Buffer.from(bytes).toString(ENCODING_TYPE.BASE64); + } + const bytes = await fs.promises.readFile(request.filePath as string); + return bytes.toString(ENCODING_TYPE.BASE64); + } catch (_e) { + printLog( + parameterizedString(logs.errorLogs.FAILED_TO_ENCODE_FILE_IN_DEIDENTIFY_FILE, DETECT_INTERFACE_NAME), + MessageType.ERROR, + logLevel, + ); + throw new SkyflowError(SKYFLOW_ERROR_CODE.FAILED_TO_ENCODE_FILE); + } +} + +/** The format matching the file extension, or undefined when unknown so the server infers it. @internal */ +export function inferFormat(fileName: string | null): FileDataFormat | undefined { + if (!fileName) { + return undefined; + } + const ext = path.extname(fileName).replace(/^\./, "").toUpperCase(); + return ext && Object.prototype.hasOwnProperty.call(FileDataFormat, ext) + ? (FileDataFormat as Record)[ext] + : undefined; +} + +/** + * Builds the submit request. For local files this reads and base64-encodes the file and infers the + * format from its name when none was given. + * @internal + */ +export async function toDeidentifyFileRequest( + request: DeidentifyFileRequest, + vaultId: string, + logLevel: LogLevel = LogLevel.ERROR, +): Promise { + const isLocal = !!request.file || !!request.filePath; + let value = request.value as string; + let format = request.dataFormat; + if (isLocal) { + value = await encodeLocalFile(request, logLevel); + if (!format) { + format = inferFormat(localFileName(request)); + } + } + // A local file is always sent as BASE64, whatever dataSource the request was left with. + const dataSource = isLocal ? BASE64_SOURCE : (request.dataSource as string); + const apiRequest: DeidentifyFileRequestV2 = { + dataSource: dataSource as DeidentifyFileRequestV2DataSource, + value, + }; + if (format) { + // The wire value for a file format is the lower-case extension. + apiRequest.dataFormat = format.toLowerCase() as DeidentifyFileRequestV2DataFormat; + } + if (request.configurationId !== undefined) { + apiRequest.configurationId = request.configurationId; + } + if (sendsConfiguration(request.configurationId, request.configuration)) { + apiRequest.configuration = toDetectConfig(request.configuration, vaultId); + } + return apiRequest; +} + +/** @internal */ +export function toReidentifyStringRequest(request: ReidentifyStringRequest, vaultId: string): ReidentifyStringRequestV2 { + const apiRequest: ReidentifyStringRequestV2 = { vaultId, text: request.text }; + if (request.redactionLevel) { + apiRequest.redactionLevel = toRedactionLevels(request.redactionLevel); + } + return apiRequest; +} + +/** @internal */ +export function toReidentifyFileRequest(request: ReidentifyFileRequest, vaultId: string): ReidentifyFileRequestV2 { + const apiRequest: ReidentifyFileRequestV2 = { + dataSource: request.dataSource as ReidentifyFileRequestV2DataSource, + value: request.value, + vaultId, + }; + if (request.dataFormat) { + // The wire value for a file format is the lower-case extension. + apiRequest.dataFormat = request.dataFormat.toLowerCase() as ReidentifyFileRequestV2DataFormat; + } + if (request.redactionLevel) { + apiRequest.redactionLevel = toRedactionLevels(request.redactionLevel); + } + return apiRequest; +} + +/** @internal */ +export function toCheckGuardrailsRequest(request: CheckGuardrailsRequest, vaultId: string): DetectGuardrailsRequestV2 { + const apiRequest: DetectGuardrailsRequestV2 = { text: request.text, vaultId }; + if (request.checkToxicity !== undefined) { + apiRequest.checkToxicity = request.checkToxicity; + } + if (request.denyTopics !== undefined) { + apiRequest.denyTopics = request.denyTopics; + } + return apiRequest; +} + +/** + * @param configuration the inline configuration, or undefined to send a configuration that carries + * only the vault id + * @internal + */ +export function toDetectConfig(configuration: DetectConfiguration | undefined, vaultId: string): DetectConfigV2 { + const config: DetectConfigV2 = { vaultId }; + if (!configuration) { + return config; + } + if (configuration.name !== undefined) { + config.name = configuration.name; + } + if (configuration.description !== undefined) { + config.description = configuration.description; + } + if (configuration.detect) { + config.detect = toDetect(configuration.detect); + } + if (configuration.fileMapping) { + config.fileMapping = toFileMappings(configuration.fileMapping); + } + if (configuration.media) { + config.media = toMedia(configuration.media); + } + return config; +} + +function compact(value: T): T { + // Drop undefined keys so the wire body only carries what the caller set. + const out = {} as T; + Object.keys(value).forEach((key) => { + const v = (value as Record)[key]; + if (v !== undefined) { + (out as Record)[key] = v; + } + }); + return out; +} + +function toDetect(detect: Detect): DetectV2 { + return compact({ + entities: detect.entities?.filter((e) => e != null).map(toEntity), + objectEntities: detect.objectEntities?.filter((e) => e != null).map(toObjectEntity), + restrict: detect.restrict, + skip: detect.skip, + customTypes: detect.customTypes?.filter((c) => c != null).map(toCustomType), + returnEntities: detect.returnEntities, + }); +} + +function toEntity(entity: Entity): DetectEntity { + return compact({ + entityType: entity.entityType, + deidentificationType: entity.deidentificationType, + destination: entity.destination, + transformation: entity.transformation ? toTransformation(entity.transformation) : undefined, + }); +} + +function toTransformation(transformation: Transformation): EntityTransformation { + const shiftDates = transformation.shiftDates; + return compact({ + shiftDates: shiftDates ? compact({ minDays: shiftDates.minDays, maxDays: shiftDates.maxDays }) : undefined, + }); +} + +function toCustomType(customType: CustomType): DetectCustomTypes { + return compact({ label: customType.label, match: customType.match }); +} + +function toObjectEntity(objectEntity: ObjectEntity): DetectObjectEntity { + return compact({ + entityType: objectEntity.entityType, + deidentificationType: objectEntity.deidentificationType, + }); +} + +function toFileMappings(mappings: FileMapping[]): FileMappingV2[] { + return mappings + .filter((m) => m != null) + .map((m) => + compact({ + source: m.source, + destination: m.destination, + outputFileNameSuffix: m.outputFileNameSuffix, + entitiesDestination: m.entitiesDestination, + objectEntitiesDestination: m.objectEntitiesDestination, + }), + ); +} + +function toMedia(media: Media): MediaV2 { + return compact({ + audio: media.audio ? toAudio(media.audio) : undefined, + document: media.document ? toDocument(media.document) : undefined, + image: media.image ? toImage(media.image) : undefined, + }); +} + +function toAudio(audio: Audio): AudioV2 { + const bleep = audio.bleep; + return compact({ + outputProcessedAudio: audio.outputProcessedAudio, + outputTranscription: audio.outputTranscription, + bleep: bleep + ? compact({ + startPadding: bleep.startPadding, + stopPadding: bleep.stopPadding, + frequency: bleep.frequency, + gain: bleep.gain, + }) + : undefined, + }); +} + +function toDocument(document: Document): DocumentV2 { + const pdf = document.pdf; + return compact({ + pdf: pdf + ? compact({ processingMode: pdf.processingMode, density: pdf.density, maxResolution: pdf.maxResolution }) + : undefined, + }); +} + +function toImage(image: Image): ImageV2 { + return compact({ + outputProcessedImage: image.outputProcessedImage, + outputOcrText: image.outputOcrText, + maskingMethod: image.maskingMethod, + }); +} + +/** @internal */ +export function toRedactionLevels(levels: RedactionRule[]): RedactionLevelV2[] { + return levels + .filter((l) => l != null) + .map((l) => + compact({ + entityName: l.entityName, + redactionType: l.redactionType, + tokenGroupName: l.tokenGroupName, + redactionPattern: l.redactionPattern, + }), + ); +} diff --git a/common/src/detect/internal/response-mapper.ts b/common/src/detect/internal/response-mapper.ts new file mode 100644 index 00000000..c21e91da --- /dev/null +++ b/common/src/detect/internal/response-mapper.ts @@ -0,0 +1,182 @@ +// Converts generated V2 API responses into public SDK response models. Internal to the SDK. +import { + DeidentifyStringResponseV2, + DeidentifyStringResponseV2DetectedEntity, + DeidentifyStringResponseV2EntityLocation, + DetectGuardrailsResponseV2, + DetectRunsResponseV2, + FileOutput as FileOutputV2, + Metrics as MetricsV2, + ReidentifyFileResponseV2, + ReidentifyStringResponseV2, +} from "../../_generated_/rest/api"; +import { + DeidentifyFileSource, + DetectRunStatus, + DetectedEntity, + EntityLocation, + FileDataFormat, + FileOutput, + GuardrailsValidation, + Metrics, + ReidentifyFileSource, +} from "../types"; +import DeidentifyStringResponse from "../model/response/deidentify-string"; +import ReidentifyStringResponse from "../model/response/reidentify-string"; +import DeidentifyFileResponse from "../model/response/deidentify-file"; +import ReidentifyFileResponse from "../model/response/reidentify-file"; +import GetRunResponse from "../model/response/get-run"; +import CheckGuardrailsResponse from "../model/response/check-guardrails"; + +function orNull(value: T | undefined | null): T | null { + return value === undefined || value === null ? null : value; +} + +/** + * Resolves a wire output type onto the SDK's own source enum; `null` for a value the SDK does not + * know. Each SDK's DetectController supplies one per source enum. @internal + */ +export type SourceResolver = (value: string | null | undefined) => S | null; + +/** Maps a wire value onto a public enum, tolerating values this SDK version does not know. @internal */ +export function enumOrDefault( + values: Record, + value: unknown, + fallback: E | null, +): E | null { + if (value === undefined || value === null) { + return fallback; + } + const match = Object.values(values).find((v) => v === String(value)); + return match === undefined ? fallback : match; +} + +/** @internal */ +export function toMetrics(metrics: MetricsV2 | undefined | null): Metrics | null { + if (!metrics) { + return null; + } + return { + size: orNull(metrics.size), + wordCount: orNull(metrics.wordCount), + characterCount: orNull(metrics.characterCount), + slides: orNull(metrics.slides), + pages: orNull(metrics.pages), + duration: orNull(metrics.duration), + }; +} + +function toEntityLocation(location: DeidentifyStringResponseV2EntityLocation | undefined | null): EntityLocation | null { + if (!location) { + return null; + } + return { + startIndex: orNull(location.startIndex), + endIndex: orNull(location.endIndex), + startIndexProcessed: orNull(location.startIndexProcessed), + endIndexProcessed: orNull(location.endIndexProcessed), + }; +} + +function toDetectedEntity(entity: DeidentifyStringResponseV2DetectedEntity): DetectedEntity { + return { + token: orNull(entity.token), + value: orNull(entity.value), + location: toEntityLocation(entity.location), + entityType: orNull(entity.entityType), + entityScores: entity.entityScores ? { ...entity.entityScores } : {}, + }; +} + +/** @internal */ +export function toDeidentifyStringResponse(response: DeidentifyStringResponseV2): DeidentifyStringResponse { + const entities = (response.entities ?? []).filter((e) => e != null).map(toDetectedEntity); + return new DeidentifyStringResponse({ + processedText: orNull(response.processedText), + entities, + metrics: toMetrics(response.metrics), + }); +} + +/** @internal */ +export function toReidentifyStringResponse(response: ReidentifyStringResponseV2): ReidentifyStringResponse { + return new ReidentifyStringResponse({ + processedText: orNull(response.processedText), + metrics: toMetrics(response.metrics), + }); +} + +/** @internal */ +export function toFileOutput(output: FileOutputV2): FileOutput { + return { + processedFile: orNull(output.processedFile), + processedFileType: output.processedFileType === undefined || output.processedFileType === null ? null : String(output.processedFileType), + processedFileExtension: enumOrDefault( + FileDataFormat, + output.processedFileExtension === undefined || output.processedFileExtension === null + ? undefined + : String(output.processedFileExtension).toUpperCase(), + null, + ), + }; +} + +function toFileOutputs(outputs: FileOutputV2[] | undefined | null): FileOutput[] { + return (outputs ?? []).filter((o) => o != null).map(toFileOutput); +} + +/** @internal */ +export function toGetRunResponse( + runId: string, + response: DetectRunsResponseV2, + resolveSource: SourceResolver, +): GetRunResponse { + return new GetRunResponse({ + runId, + status: enumOrDefault(DetectRunStatus, response.status, DetectRunStatus.UNKNOWN) as DetectRunStatus, + outputType: resolveSource(response.outputType === undefined || response.outputType === null ? null : String(response.outputType)), + output: toFileOutputs(response.output), + metrics: toMetrics(response.metrics), + message: orNull(response.message), + }); +} + +/** Response for a submit-only call: run id and nothing else. @internal */ +export function submittedDeidentifyFileResponse(runId: string | null): DeidentifyFileResponse { + return new DeidentifyFileResponse({ runId, status: null, outputType: null, output: [], metrics: null, message: null }); +} + +/** Merges the submit's run id with the polled run. @internal */ +export function toDeidentifyFileResponse(runId: string, run: GetRunResponse): DeidentifyFileResponse { + return new DeidentifyFileResponse({ + runId, + status: run.status, + outputType: run.outputType, + output: run.output, + metrics: run.metrics, + message: run.message, + }); +} + +/** @internal */ +export function toReidentifyFileResponse( + response: ReidentifyFileResponseV2, + resolveSource: SourceResolver, +): ReidentifyFileResponse { + return new ReidentifyFileResponse({ + status: enumOrDefault(DetectRunStatus, response.status, DetectRunStatus.UNKNOWN) as DetectRunStatus, + outputType: resolveSource(response.outputType === undefined || response.outputType === null ? null : String(response.outputType)), + output: toFileOutputs(response.output), + metrics: toMetrics(response.metrics), + }); +} + +/** @internal */ +export function toCheckGuardrailsResponse(response: DetectGuardrailsResponseV2): CheckGuardrailsResponse { + return new CheckGuardrailsResponse({ + text: orNull(response.text), + validation: enumOrDefault(GuardrailsValidation, response.validation, GuardrailsValidation.UNKNOWN) as GuardrailsValidation, + toxic: orNull(response.toxic), + deniedTopic: orNull(response.deniedTopic), + }); +} diff --git a/common/src/detect/internal/run-poller.ts b/common/src/detect/internal/run-poller.ts new file mode 100644 index 00000000..bcd4cf75 --- /dev/null +++ b/common/src/detect/internal/run-poller.ts @@ -0,0 +1,95 @@ +// Polls a Detect file run until it reaches a terminal status or the PollOptions budget is used up. +// Shared by every SDK: the caller supplies how a run is fetched, so each controller plugs in its +// own `getRun`. +// +// Backoff is exponential, 1, 2, 4, 8, 16 then 32 seconds between lookups, with the last sleep +// trimmed so the total never exceeds `waitTime`. The first lookup happens immediately after the +// submit. SUCCESS, FAILED and UNKNOWN are terminal; every other status keeps polling. +import { LogLevel, MessageType, parameterizedString, printLog } from "../../utils"; +import logs from "../../utils/logs"; +import { DetectRunStatus, POLL_OPTIONS, PollOptions } from "../types"; +import GetRunResponse from "../model/response/get-run"; + +/** Fetches the current state of a run. @internal */ +export type RunFetcher = (runId: string) => Promise; + +/** Pauses for the given number of milliseconds. Replaceable in tests. @internal */ +export type Sleeper = (millis: number) => Promise; + +const defaultSleeper: Sleeper = (millis) => new Promise((resolve) => setTimeout(resolve, millis)); + +const TERMINAL = new Set([DetectRunStatus.SUCCESS, DetectRunStatus.FAILED, DetectRunStatus.UNKNOWN]); + +/** @internal */ +export function effectiveWaitTime(options: PollOptions | undefined): number { + return options?.waitTime ?? POLL_OPTIONS.DEFAULT_WAIT_TIME_SECONDS; +} + +/** @internal */ +export function effectiveMaxAttempts(options: PollOptions | undefined): number { + return options?.maxAttempts ?? POLL_OPTIONS.DEFAULT_MAX_ATTEMPTS; +} + +/** + * @param runId run to poll + * @param options wait time and attempt limit, defaults applied + * @param fetcher how to look the run up + * @returns the terminal run, or the last non-terminal run seen when the budget ran out + * @throws SkyflowError if a lookup fails + * @internal + */ +export async function pollRun( + runId: string, + options: PollOptions | undefined, + fetcher: RunFetcher, + logLevel: LogLevel = LogLevel.ERROR, + sleeper: Sleeper = defaultSleeper, +): Promise { + const waitTime = effectiveWaitTime(options); + const maxAttempts = effectiveMaxAttempts(options); + printLog( + parameterizedString(logs.infoLogs.POLL_RUN_STARTED, runId, String(waitTime), String(maxAttempts)), + MessageType.LOG, + logLevel, + ); + + let elapsedSeconds = 0; + let delaySeconds = 1; + let attempts = 0; + for (;;) { + let run = await fetcher(runId); + attempts += 1; + if (!run || !run.status) { + // A fetcher that reports nothing is treated as an unknown, terminal state. + run = new GetRunResponse({ + runId, + status: DetectRunStatus.UNKNOWN, + outputType: null, + output: [], + metrics: null, + message: null, + }); + } + const status = run.status as DetectRunStatus; + if (TERMINAL.has(status)) { + printLog( + parameterizedString(logs.infoLogs.POLL_RUN_TERMINAL, runId, status, String(attempts)), + MessageType.LOG, + logLevel, + ); + return run; + } + if (attempts >= maxAttempts || elapsedSeconds >= waitTime) { + printLog( + parameterizedString(logs.infoLogs.POLL_RUN_BUDGET_EXHAUSTED, runId, status, String(attempts), String(elapsedSeconds)), + MessageType.LOG, + logLevel, + ); + return run; + } + const sleepSeconds = Math.min(delaySeconds, waitTime - elapsedSeconds); + await sleeper(sleepSeconds * 1000); + elapsedSeconds += sleepSeconds; + delaySeconds = Math.min(delaySeconds * 2, POLL_OPTIONS.MAX_DELAY_SECONDS); + } +} diff --git a/common/src/detect/internal/validations.ts b/common/src/detect/internal/validations.ts new file mode 100644 index 00000000..3a8d72a5 --- /dev/null +++ b/common/src/detect/internal/validations.ts @@ -0,0 +1,165 @@ +// Client-side validation for Detect V2 requests. Internal to the SDK. +import * as fs from "fs"; +import { LogLevel, MessageType, parameterizedString, printLog } from "../../utils"; +import logs from "../../utils/logs"; +import SkyflowError from "../../error"; +import SKYFLOW_ERROR_CODE from "../../error/codes"; +import { POLL_OPTIONS, PollOptions } from "../types"; +import DeidentifyStringRequest from "../model/request/deidentify-string"; +import DeidentifyFileRequest from "../model/request/deidentify-file"; +import ReidentifyStringRequest from "../model/request/reidentify-string"; +import ReidentifyFileRequest from "../model/request/reidentify-file"; +import GetRunRequest from "../model/request/get-run"; +import CheckGuardrailsRequest from "../model/request/check-guardrails"; + +/** @internal */ +export const DETECT_INTERFACE_NAME = "detect"; + +function isBlank(value: unknown): boolean { + return typeof value !== "string" || value.trim().length === 0; +} + +/** Looks like a web `File`: something that can be read as an ArrayBuffer. Callers pass a non-null value. */ +function isFileLike(value: unknown): boolean { + const candidate = value as { arrayBuffer?: unknown }; + return candidate.arrayBuffer instanceof Function; +} + +function invalid(log: string, code: { httpCode: number; message: string }, logLevel: LogLevel): SkyflowError { + printLog(parameterizedString(log, DETECT_INTERFACE_NAME), MessageType.ERROR, logLevel); + return new SkyflowError(code); +} + +/** @internal */ +export function validateDeidentifyStringRequest(request: DeidentifyStringRequest, logLevel: LogLevel = LogLevel.ERROR): void { + if (!request) { + throw invalid(logs.errorLogs.DEIDENTIFY_STRING_REQUEST_NULL, SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_STRING_REQUEST, logLevel); + } + if (isBlank(request.text)) { + throw invalid(logs.errorLogs.INVALID_TEXT_IN_DEIDENTIFY_STRING, SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_DEIDENTIFY_STRING, logLevel); + } + // configurationId and configuration are both optional on the API and are passed through as given; + // with neither set the mapper sends a configuration carrying only the vault id. +} + +/** @internal */ +export function validateDeidentifyFileRequest(request: DeidentifyFileRequest, logLevel: LogLevel = LogLevel.ERROR): void { + if (!request) { + throw invalid(logs.errorLogs.DEIDENTIFY_FILE_REQUEST_NULL, SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST, logLevel); + } + // The file comes from exactly one of filePath, file or value; dataSource belongs to value only. + const hasFile = request.file !== undefined && request.file !== null; + const hasFilePath = request.filePath !== undefined && request.filePath !== null; + const hasValue = request.value !== undefined && request.value !== null; + const sources = [hasFile, hasFilePath, hasValue].filter(Boolean).length; + if (sources === 0) { + throw invalid(logs.errorLogs.MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE, SKYFLOW_ERROR_CODE.MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE, logLevel); + } + if (sources > 1) { + throw invalid(logs.errorLogs.MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE, SKYFLOW_ERROR_CODE.MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE, logLevel); + } + if (hasFile) { + if (!isFileLike(request.file)) { + throw invalid(logs.errorLogs.INVALID_FILE_IN_DEIDENTIFY_FILE, SKYFLOW_ERROR_CODE.INVALID_FILE_IN_DEIDENTIFY_FILE, logLevel); + } + } else if (hasFilePath) { + let isFile = false; + try { + isFile = fs.statSync(request.filePath as string).isFile(); + } catch (_e) { + isFile = false; + } + if (!isFile) { + throw invalid(logs.errorLogs.FILE_NOT_FOUND_IN_DEIDENTIFY_FILE, SKYFLOW_ERROR_CODE.FILE_NOT_FOUND_IN_DEIDENTIFY_FILE, logLevel); + } + } else { + if (!request.dataSource) { + throw invalid(logs.errorLogs.INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE, SKYFLOW_ERROR_CODE.INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE, logLevel); + } + if (isBlank(request.value)) { + throw invalid(logs.errorLogs.INVALID_VALUE_IN_DEIDENTIFY_FILE, SKYFLOW_ERROR_CODE.INVALID_VALUE_IN_DEIDENTIFY_FILE, logLevel); + } + } + // configurationId and configuration are both optional on the API and are passed through as given. + validatePollOptions(request.pollOptions, logLevel); + if (!request.pollOptions && !isBlank(request.outputDirectory)) { + throw invalid(logs.errorLogs.OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS, SKYFLOW_ERROR_CODE.OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS, logLevel); + } +} + +/** @internal */ +export function validatePollOptions(options: PollOptions | undefined, logLevel: LogLevel = LogLevel.ERROR): void { + if (!options) { + return; + } + const { waitTime, maxAttempts } = options; + if ( + waitTime !== undefined && + (typeof waitTime !== "number" || !Number.isFinite(waitTime) || waitTime < 1 || waitTime > POLL_OPTIONS.MAX_WAIT_TIME_SECONDS) + ) { + throw invalid(logs.errorLogs.INVALID_WAIT_TIME_IN_POLL_OPTIONS, SKYFLOW_ERROR_CODE.INVALID_WAIT_TIME_IN_POLL_OPTIONS, logLevel); + } + if ( + maxAttempts !== undefined && + (typeof maxAttempts !== "number" || !Number.isFinite(maxAttempts) || maxAttempts < 1 || maxAttempts > POLL_OPTIONS.MAX_MAX_ATTEMPTS) + ) { + throw invalid(logs.errorLogs.INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS, SKYFLOW_ERROR_CODE.INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS, logLevel); + } +} + +function validateRedactionLevel(redactionLevel: unknown, logLevel: LogLevel): void { + if (redactionLevel !== undefined && redactionLevel !== null && !Array.isArray(redactionLevel)) { + throw invalid(logs.errorLogs.INVALID_REDACTION_LEVEL, SKYFLOW_ERROR_CODE.INVALID_REDACTION_LEVEL, logLevel); + } +} + +/** @internal */ +export function validateReidentifyStringRequest(request: ReidentifyStringRequest, logLevel: LogLevel = LogLevel.ERROR): void { + if (!request) { + throw invalid(logs.errorLogs.REIDENTIFY_STRING_REQUEST_NULL, SKYFLOW_ERROR_CODE.INVALID_REIDENTIFY_STRING_REQUEST, logLevel); + } + if (isBlank(request.text)) { + throw invalid(logs.errorLogs.INVALID_TEXT_IN_REIDENTIFY_STRING, SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_REIDENTIFY_STRING, logLevel); + } + // redactionLevel entries are optional on the API and are passed through as given. + validateRedactionLevel(request.redactionLevel, logLevel); +} + +/** @internal */ +export function validateReidentifyFileRequest(request: ReidentifyFileRequest, logLevel: LogLevel = LogLevel.ERROR): void { + if (!request) { + throw invalid(logs.errorLogs.REIDENTIFY_FILE_REQUEST_NULL, SKYFLOW_ERROR_CODE.INVALID_REIDENTIFY_FILE_REQUEST, logLevel); + } + if (!request.dataSource) { + throw invalid(logs.errorLogs.INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE, SKYFLOW_ERROR_CODE.INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE, logLevel); + } + if (isBlank(request.value)) { + throw invalid(logs.errorLogs.INVALID_VALUE_IN_REIDENTIFY_FILE, SKYFLOW_ERROR_CODE.INVALID_VALUE_IN_REIDENTIFY_FILE, logLevel); + } + validateRedactionLevel(request.redactionLevel, logLevel); +} + +/** @internal */ +export function validateGetRunRequest(request: GetRunRequest, logLevel: LogLevel = LogLevel.ERROR): void { + if (!request) { + throw invalid(logs.errorLogs.GET_RUN_REQUEST_NULL, SKYFLOW_ERROR_CODE.INVALID_GET_RUN_REQUEST, logLevel); + } + if (isBlank(request.runId)) { + throw invalid(logs.errorLogs.INVALID_RUN_ID_IN_GET_RUN, SKYFLOW_ERROR_CODE.INVALID_RUN_ID_IN_GET_RUN, logLevel); + } +} + +/** @internal */ +export function validateCheckGuardrailsRequest(request: CheckGuardrailsRequest, logLevel: LogLevel = LogLevel.ERROR): void { + if (!request) { + throw invalid(logs.errorLogs.CHECK_GUARDRAILS_REQUEST_NULL, SKYFLOW_ERROR_CODE.INVALID_CHECK_GUARDRAILS_REQUEST, logLevel); + } + if (isBlank(request.text)) { + throw invalid(logs.errorLogs.INVALID_TEXT_IN_CHECK_GUARDRAILS, SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_CHECK_GUARDRAILS, logLevel); + } + const topics = request.denyTopics; + if (topics !== undefined && topics !== null && (!Array.isArray(topics) || topics.some((t) => typeof t !== "string"))) { + throw invalid(logs.errorLogs.INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS, SKYFLOW_ERROR_CODE.INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS, logLevel); + } + // checkToxicity is optional on the API and is passed through as given. +} diff --git a/common/src/detect/model/request/check-guardrails/index.ts b/common/src/detect/model/request/check-guardrails/index.ts new file mode 100644 index 00000000..823c5595 --- /dev/null +++ b/common/src/detect/model/request/check-guardrails/index.ts @@ -0,0 +1,49 @@ +/** Optional fields of a {@link CheckGuardrailsRequest}. */ +export interface CheckGuardrailsRequestFields { + /** Whether to check for toxicity. Sent only when set. */ + checkToxicity?: boolean; + /** Topics to deny. Sent only when set. */ + denyTopics?: string[]; +} + +/** Request for `detect().checkGuardrails()`. */ +class CheckGuardrailsRequest { + private _text: string; + + private _checkToxicity?: boolean; + + private _denyTopics?: string[]; + + constructor(text: string, fields?: CheckGuardrailsRequestFields) { + this._text = text; + this._checkToxicity = fields?.checkToxicity; + this._denyTopics = fields?.denyTopics; + } + + /** Text to screen. */ + get text(): string { + return this._text; + } + + set text(value: string) { + this._text = value; + } + + get checkToxicity(): boolean | undefined { + return this._checkToxicity; + } + + set checkToxicity(value: boolean | undefined) { + this._checkToxicity = value; + } + + get denyTopics(): string[] | undefined { + return this._denyTopics; + } + + set denyTopics(value: string[] | undefined) { + this._denyTopics = value; + } +} + +export default CheckGuardrailsRequest; diff --git a/common/src/detect/model/request/deidentify-file/index.ts b/common/src/detect/model/request/deidentify-file/index.ts new file mode 100644 index 00000000..036a7a90 --- /dev/null +++ b/common/src/detect/model/request/deidentify-file/index.ts @@ -0,0 +1,142 @@ +import { DeidentifyFileSource, DetectConfiguration, FileDataFormat, PollOptions } from "../../../types"; + +/** + * Fields of a {@link DeidentifyFileRequest}. Supply the file through exactly + * one of `value` (with a `dataSource`), `file` or `filePath`. + */ +export interface DeidentifyFileRequestFields { + /** How `value` supplies the file: a value of your SDK's `DeidentifySourceType` (`BASE64` or `PRESIGNED_URL`; skyvault also accepts `SKYFLOW_ID`). */ + dataSource?: DeidentifyFileSource; + /** Base64 content, a presigned URL or, on skyvault, a skyflow id, per `dataSource`. */ + value?: string; + /** An in-memory `File`; read and base64-encoded when sent. Requires Node.js 20+. */ + file?: File; + /** Path to a local file; read and base64-encoded when sent. */ + filePath?: string; + /** File format. Inferred from the file name's extension when omitted for a local file. */ + dataFormat?: FileDataFormat; + /** + * Id of a Detect configuration created through the management API. + * Sending it together with `configuration` is rejected by the API. + */ + configurationId?: string; + /** + * Inline configuration. With neither this nor `configurationId` set, the + * SDK sends only the vault id and the server applies the vault's defaults. + */ + configuration?: DetectConfiguration; + /** Poll the run after submitting. Omit to submit only and return the run id. */ + pollOptions?: PollOptions; + /** + * Where outputs are written on success. Requires `pollOptions` and a local + * file (`file` or `filePath`). Defaults to the working directory. + */ + outputDirectory?: string; +} + +/** Request for `detect().deidentifyFile()`. */ +class DeidentifyFileRequest { + private _dataSource?: DeidentifyFileSource; + + private _value?: string; + + private _file?: File; + + private _filePath?: string; + + private _dataFormat?: FileDataFormat; + + private _configurationId?: string; + + private _configuration?: DetectConfiguration; + + private _pollOptions?: PollOptions; + + private _outputDirectory?: string; + + constructor(fields: DeidentifyFileRequestFields = {}) { + this._dataSource = fields.dataSource; + this._value = fields.value; + this._file = fields.file; + this._filePath = fields.filePath; + this._dataFormat = fields.dataFormat; + this._configurationId = fields.configurationId; + this._configuration = fields.configuration; + this._pollOptions = fields.pollOptions; + this._outputDirectory = fields.outputDirectory; + } + + get dataSource(): DeidentifyFileSource | undefined { + return this._dataSource; + } + + set dataSource(value: DeidentifyFileSource | undefined) { + this._dataSource = value; + } + + get value(): string | undefined { + return this._value; + } + + set value(value: string | undefined) { + this._value = value; + } + + get file(): File | undefined { + return this._file; + } + + set file(value: File | undefined) { + this._file = value; + } + + get filePath(): string | undefined { + return this._filePath; + } + + set filePath(value: string | undefined) { + this._filePath = value; + } + + get dataFormat(): FileDataFormat | undefined { + return this._dataFormat; + } + + set dataFormat(value: FileDataFormat | undefined) { + this._dataFormat = value; + } + + get configurationId(): string | undefined { + return this._configurationId; + } + + set configurationId(value: string | undefined) { + this._configurationId = value; + } + + get configuration(): DetectConfiguration | undefined { + return this._configuration; + } + + set configuration(value: DetectConfiguration | undefined) { + this._configuration = value; + } + + get pollOptions(): PollOptions | undefined { + return this._pollOptions; + } + + set pollOptions(value: PollOptions | undefined) { + this._pollOptions = value; + } + + get outputDirectory(): string | undefined { + return this._outputDirectory; + } + + set outputDirectory(value: string | undefined) { + this._outputDirectory = value; + } +} + +export default DeidentifyFileRequest; diff --git a/common/src/detect/model/request/deidentify-string/index.ts b/common/src/detect/model/request/deidentify-string/index.ts new file mode 100644 index 00000000..a433eb82 --- /dev/null +++ b/common/src/detect/model/request/deidentify-string/index.ts @@ -0,0 +1,57 @@ +import { DetectConfiguration } from "../../../types"; + +/** Optional fields of a {@link DeidentifyStringRequest}. */ +export interface DeidentifyStringRequestFields { + /** + * Id of a Detect configuration created through the management API. + * Sending it together with `configuration` is rejected by the API. + */ + configurationId?: string; + /** + * Inline configuration. With neither this nor `configurationId` set, the + * SDK sends only the vault id and the server applies the vault's defaults. + */ + configuration?: DetectConfiguration; +} + +/** Request for `detect().deidentifyString()`. */ +class DeidentifyStringRequest { + private _text: string; + + private _configurationId?: string; + + private _configuration?: DetectConfiguration; + + constructor(text: string, fields?: DeidentifyStringRequestFields) { + this._text = text; + this._configurationId = fields?.configurationId; + this._configuration = fields?.configuration; + } + + /** Text to de-identify. */ + get text(): string { + return this._text; + } + + set text(value: string) { + this._text = value; + } + + get configurationId(): string | undefined { + return this._configurationId; + } + + set configurationId(value: string | undefined) { + this._configurationId = value; + } + + get configuration(): DetectConfiguration | undefined { + return this._configuration; + } + + set configuration(value: DetectConfiguration | undefined) { + this._configuration = value; + } +} + +export default DeidentifyStringRequest; diff --git a/common/src/detect/model/request/get-run/index.ts b/common/src/detect/model/request/get-run/index.ts new file mode 100644 index 00000000..e6a5bb31 --- /dev/null +++ b/common/src/detect/model/request/get-run/index.ts @@ -0,0 +1,19 @@ +/** Request for `detect().getRun()`. */ +class GetRunRequest { + private _runId: string; + + constructor(runId: string) { + this._runId = runId; + } + + /** The run id returned by a prior `deidentifyFile` call. */ + get runId(): string { + return this._runId; + } + + set runId(value: string) { + this._runId = value; + } +} + +export default GetRunRequest; diff --git a/common/src/detect/model/request/reidentify-file/index.ts b/common/src/detect/model/request/reidentify-file/index.ts new file mode 100644 index 00000000..adbd2320 --- /dev/null +++ b/common/src/detect/model/request/reidentify-file/index.ts @@ -0,0 +1,66 @@ +import { FileDataFormat, RedactionRule, ReidentifyFileSource } from "../../../types"; + +/** Optional fields of a {@link ReidentifyFileRequest}. */ +export interface ReidentifyFileRequestFields { + /** + * File format. Limited to the formats that support `VAULT_TOKEN`: `TXT`, + * `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` or `PPTX`. + */ + dataFormat?: FileDataFormat; + /** Redaction rules, as your SDK's `RedactionLevel`. Without rules every token is replaced by its original value. */ + redactionLevel?: RedactionRule[]; +} + +/** Request for `detect().reidentifyFile()`. Synchronous: the re-identified file comes back in the response. */ +class ReidentifyFileRequest { + private _dataSource: ReidentifyFileSource; + + private _value: string; + + private _dataFormat?: FileDataFormat; + + private _redactionLevel?: RedactionRule[]; + + constructor(dataSource: ReidentifyFileSource, value: string, fields?: ReidentifyFileRequestFields) { + this._dataSource = dataSource; + this._value = value; + this._dataFormat = fields?.dataFormat; + this._redactionLevel = fields?.redactionLevel; + } + + /** A value of your SDK's `ReidentifySourceType`: `BASE64` or `PRESIGNED_URL`. Re-identification does not accept a skyflow id. */ + get dataSource(): ReidentifyFileSource { + return this._dataSource; + } + + set dataSource(value: ReidentifyFileSource) { + this._dataSource = value; + } + + /** Base64 content or a presigned URL, per `dataSource`. */ + get value(): string { + return this._value; + } + + set value(value: string) { + this._value = value; + } + + get dataFormat(): FileDataFormat | undefined { + return this._dataFormat; + } + + set dataFormat(value: FileDataFormat | undefined) { + this._dataFormat = value; + } + + get redactionLevel(): RedactionRule[] | undefined { + return this._redactionLevel; + } + + set redactionLevel(value: RedactionRule[] | undefined) { + this._redactionLevel = value; + } +} + +export default ReidentifyFileRequest; diff --git a/common/src/detect/model/request/reidentify-string/index.ts b/common/src/detect/model/request/reidentify-string/index.ts new file mode 100644 index 00000000..b4bb1f59 --- /dev/null +++ b/common/src/detect/model/request/reidentify-string/index.ts @@ -0,0 +1,38 @@ +import { RedactionRule } from "../../../types"; + +/** Optional fields of a {@link ReidentifyStringRequest}. */ +export interface ReidentifyStringRequestFields { + /** Redaction rules, as your SDK's `RedactionLevel`. Without rules every token is replaced by its original value. */ + redactionLevel?: RedactionRule[]; +} + +/** Request for `detect().reidentifyString()`. */ +class ReidentifyStringRequest { + private _text: string; + + private _redactionLevel?: RedactionRule[]; + + constructor(text: string, fields?: ReidentifyStringRequestFields) { + this._text = text; + this._redactionLevel = fields?.redactionLevel; + } + + /** The de-identified text to re-identify. */ + get text(): string { + return this._text; + } + + set text(value: string) { + this._text = value; + } + + get redactionLevel(): RedactionRule[] | undefined { + return this._redactionLevel; + } + + set redactionLevel(value: RedactionRule[] | undefined) { + this._redactionLevel = value; + } +} + +export default ReidentifyStringRequest; diff --git a/common/src/detect/model/response/check-guardrails/index.ts b/common/src/detect/model/response/check-guardrails/index.ts new file mode 100644 index 00000000..8c5e0d80 --- /dev/null +++ b/common/src/detect/model/response/check-guardrails/index.ts @@ -0,0 +1,30 @@ +import { GuardrailsValidation } from "../../../types"; + +/** Response of `detect().checkGuardrails()`. Passed through exactly as the API returned it. */ +class CheckGuardrailsResponse { + /** The screened text. */ + readonly text: string | null; + + /** `PASSED` or `FAILED`; `UNKNOWN` for a value this SDK version does not know. */ + readonly validation: GuardrailsValidation; + + /** Whether the text was judged toxic; `null` when the check did not run. */ + readonly toxic: boolean | null; + + /** Whether a denied topic was found; `null` when no topics were supplied. */ + readonly deniedTopic: boolean | null; + + constructor(input: { + text: string | null; + validation: GuardrailsValidation; + toxic: boolean | null; + deniedTopic: boolean | null; + }) { + this.text = input.text; + this.validation = input.validation; + this.toxic = input.toxic; + this.deniedTopic = input.deniedTopic; + } +} + +export default CheckGuardrailsResponse; diff --git a/common/src/detect/model/response/deidentify-file/index.ts b/common/src/detect/model/response/deidentify-file/index.ts new file mode 100644 index 00000000..014e37c5 --- /dev/null +++ b/common/src/detect/model/response/deidentify-file/index.ts @@ -0,0 +1,31 @@ +import { DeidentifyFileSource, DetectRunStatus, FileOutput, Metrics } from "../../../types"; +import { DetectRunFields } from "../get-run"; + +/** + * Response of `detect().deidentifyFile()`. Without `pollOptions` only `runId` + * is set; with polling it carries the run's status, outputs and metrics. + */ +class DeidentifyFileResponse implements DetectRunFields { + readonly runId: string | null; + + readonly status: DetectRunStatus | null; + + readonly outputType: DeidentifyFileSource | null; + + readonly output: FileOutput[]; + + readonly metrics: Metrics | null; + + readonly message: string | null; + + constructor(input: DetectRunFields) { + this.runId = input.runId; + this.status = input.status; + this.outputType = input.outputType; + this.output = input.output; + this.metrics = input.metrics; + this.message = input.message; + } +} + +export default DeidentifyFileResponse; diff --git a/common/src/detect/model/response/deidentify-string/index.ts b/common/src/detect/model/response/deidentify-string/index.ts new file mode 100644 index 00000000..a46d20c1 --- /dev/null +++ b/common/src/detect/model/response/deidentify-string/index.ts @@ -0,0 +1,21 @@ +import { DetectedEntity, Metrics } from "../../../types"; + +/** Response of `detect().deidentifyString()`. */ +class DeidentifyStringResponse { + /** The de-identified text with entity values replaced by tokens. */ + readonly processedText: string | null; + + /** Detected entities, never null. */ + readonly entities: DetectedEntity[]; + + /** Word and character counts. */ + readonly metrics: Metrics | null; + + constructor(input: { processedText: string | null; entities: DetectedEntity[]; metrics: Metrics | null }) { + this.processedText = input.processedText; + this.entities = input.entities; + this.metrics = input.metrics; + } +} + +export default DeidentifyStringResponse; diff --git a/common/src/detect/model/response/get-run/index.ts b/common/src/detect/model/response/get-run/index.ts new file mode 100644 index 00000000..f3ce0028 --- /dev/null +++ b/common/src/detect/model/response/get-run/index.ts @@ -0,0 +1,43 @@ +import { DeidentifyFileSource, DetectRunStatus, FileOutput, Metrics } from "../../../types"; + +/** Fields shared by {@link GetRunResponse} and `DeidentifyFileResponse`. */ +export interface DetectRunFields { + /** Run id, always present once the run was submitted. */ + runId: string | null; + /** Run status. `null` on a submit-only `deidentifyFile` response. */ + status: DetectRunStatus | null; + /** The run's output type as your SDK's `DeidentifySourceType`; `null` until the run completes or when the value is not one the SDK knows. */ + outputType: DeidentifyFileSource | null; + /** Every output of the run, empty until complete, never null. */ + output: FileOutput[]; + /** Size, counts, pages, slides, duration. `null` until complete. */ + metrics: Metrics | null; + /** Failure detail when the status is `FAILED`. */ + message: string | null; +} + +/** Response of `detect().getRun()`. */ +class GetRunResponse implements DetectRunFields { + readonly runId: string | null; + + readonly status: DetectRunStatus | null; + + readonly outputType: DeidentifyFileSource | null; + + readonly output: FileOutput[]; + + readonly metrics: Metrics | null; + + readonly message: string | null; + + constructor(input: DetectRunFields) { + this.runId = input.runId; + this.status = input.status; + this.outputType = input.outputType; + this.output = input.output; + this.metrics = input.metrics; + this.message = input.message; + } +} + +export default GetRunResponse; diff --git a/common/src/detect/model/response/reidentify-file/index.ts b/common/src/detect/model/response/reidentify-file/index.ts new file mode 100644 index 00000000..7d15ed54 --- /dev/null +++ b/common/src/detect/model/response/reidentify-file/index.ts @@ -0,0 +1,30 @@ +import { DetectRunStatus, FileOutput, Metrics, ReidentifyFileSource } from "../../../types"; + +/** Response of `detect().reidentifyFile()`. */ +class ReidentifyFileResponse { + /** Status of the re-identification. */ + readonly status: DetectRunStatus; + + /** The output type as your SDK's `ReidentifySourceType`; `null` when the value is not one the SDK knows. */ + readonly outputType: ReidentifyFileSource | null; + + /** Re-identified file outputs, never null. */ + readonly output: FileOutput[]; + + /** Size and counts. */ + readonly metrics: Metrics | null; + + constructor(input: { + status: DetectRunStatus; + outputType: ReidentifyFileSource | null; + output: FileOutput[]; + metrics: Metrics | null; + }) { + this.status = input.status; + this.outputType = input.outputType; + this.output = input.output; + this.metrics = input.metrics; + } +} + +export default ReidentifyFileResponse; diff --git a/common/src/detect/model/response/reidentify-string/index.ts b/common/src/detect/model/response/reidentify-string/index.ts new file mode 100644 index 00000000..2c513c1c --- /dev/null +++ b/common/src/detect/model/response/reidentify-string/index.ts @@ -0,0 +1,17 @@ +import { Metrics } from "../../../types"; + +/** Response of `detect().reidentifyString()`. */ +class ReidentifyStringResponse { + /** The re-identified text. */ + readonly processedText: string | null; + + /** Word and character counts. */ + readonly metrics: Metrics | null; + + constructor(input: { processedText: string | null; metrics: Metrics | null }) { + this.processedText = input.processedText; + this.metrics = input.metrics; + } +} + +export default ReidentifyStringResponse; diff --git a/common/src/detect/types.ts b/common/src/detect/types.ts new file mode 100644 index 00000000..fe24c4f5 --- /dev/null +++ b/common/src/detect/types.ts @@ -0,0 +1,407 @@ +// ============================================================================= +// Skyflow Detect V2 public types shared by the skyvault and flowvault SDKs. +// Mirrors the Java SDK's `com.skyflow.detect` package: the enums carry the exact +// wire values, and the configuration objects are plain interfaces so an inline +// configuration can be written as an object literal. +// ============================================================================= +import { RedactionType } from "../utils"; + +// --- Enums ------------------------------------------------------------------- + +/** Entity types Skyflow Detect can identify. `ALL` matches every type. */ +export enum EntityType { + NONE = "NONE", + ACCOUNT_NUMBER = "ACCOUNT_NUMBER", + AGE = "AGE", + ALL = "ALL", + BANK_ACCOUNT = "BANK_ACCOUNT", + BLOOD_TYPE = "BLOOD_TYPE", + CONDITION = "CONDITION", + CORPORATE_ACTION = "CORPORATE_ACTION", + CREDIT_CARD = "CREDIT_CARD", + CREDIT_CARD_EXPIRATION = "CREDIT_CARD_EXPIRATION", + CVV = "CVV", + DATE = "DATE", + DATE_INTERVAL = "DATE_INTERVAL", + DAY = "DAY", + DOB = "DOB", + DOSE = "DOSE", + DRIVER_LICENSE = "DRIVER_LICENSE", + DRUG = "DRUG", + DURATION = "DURATION", + EFFECT = "EFFECT", + EMAIL_ADDRESS = "EMAIL_ADDRESS", + EVENT = "EVENT", + FILENAME = "FILENAME", + FINANCIAL_METRIC = "FINANCIAL_METRIC", + GENDER = "GENDER", + HEALTHCARE_NUMBER = "HEALTHCARE_NUMBER", + INJURY = "INJURY", + IP_ADDRESS = "IP_ADDRESS", + LANGUAGE = "LANGUAGE", + LOCATION = "LOCATION", + LOCATION_ADDRESS = "LOCATION_ADDRESS", + LOCATION_ADDRESS_STREET = "LOCATION_ADDRESS_STREET", + LOCATION_CITY = "LOCATION_CITY", + LOCATION_COORDINATE = "LOCATION_COORDINATE", + LOCATION_COUNTRY = "LOCATION_COUNTRY", + LOCATION_STATE = "LOCATION_STATE", + LOCATION_ZIP = "LOCATION_ZIP", + MARITAL_STATUS = "MARITAL_STATUS", + MEDICAL_CODE = "MEDICAL_CODE", + MEDICAL_PROCESS = "MEDICAL_PROCESS", + MONEY = "MONEY", + MONTH = "MONTH", + NAME = "NAME", + NAME_FAMILY = "NAME_FAMILY", + NAME_GIVEN = "NAME_GIVEN", + NAME_MEDICAL_PROFESSIONAL = "NAME_MEDICAL_PROFESSIONAL", + NUMERICAL_PII = "NUMERICAL_PII", + OCCUPATION = "OCCUPATION", + ORGANIZATION = "ORGANIZATION", + ORGANIZATION_ID = "ORGANIZATION_ID", + ORGANIZATION_MEDICAL_FACILITY = "ORGANIZATION_MEDICAL_FACILITY", + ORIGIN = "ORIGIN", + PASSPORT_NUMBER = "PASSPORT_NUMBER", + PASSWORD = "PASSWORD", + PHONE_NUMBER = "PHONE_NUMBER", + PHYSICAL_ATTRIBUTE = "PHYSICAL_ATTRIBUTE", + POLITICAL_AFFILIATION = "POLITICAL_AFFILIATION", + PRODUCT = "PRODUCT", + PROJECT = "PROJECT", + RELIGION = "RELIGION", + ROUTING_NUMBER = "ROUTING_NUMBER", + SEXUALITY = "SEXUALITY", + SSN = "SSN", + STATISTICS = "STATISTICS", + TIME = "TIME", + TREND = "TREND", + URL = "URL", + USERNAME = "USERNAME", + VEHICLE_ID = "VEHICLE_ID", + YEAR = "YEAR", + ZODIAC_SIGN = "ZODIAC_SIGN", +} + +/** How a detected entity's value is replaced. */ +export enum DeidentificationType { + UNKNOWN = "UNKNOWN", + ENTITY_UNIQUE_COUNTER = "ENTITY_UNIQUE_COUNTER", + ENTITY_ONLY = "ENTITY_ONLY", + VAULT_TOKEN = "VAULT_TOKEN", + MASK = "MASK", +} + +/** + * Contract for the source of a `deidentifyFile` input and the delivery of a run's outputs. + * Each SDK exposes its own `DeidentifySourceType` enum with the values its vault supports + * (`BASE64` and `PRESIGNED_URL` on flowvault, plus `SKYFLOW_ID` on skyvault); a request takes + * that enum and a response's `outputType` is resolved to it, so an output type the SDK does + * not know comes back as `null`. + */ +export type DeidentifyFileSource = string; + +/** + * Contract for the source of a `reidentifyFile` input and the delivery of its outputs. Each + * SDK exposes its own `ReidentifySourceType` enum (`BASE64` and `PRESIGNED_URL` on both + * vaults); a response's `outputType` is resolved to it, so an unknown value comes back as `null`. + */ +export type ReidentifyFileSource = string; + +/** The wire value every SDK uses for a base64 input or output. @internal */ +export const BASE64_SOURCE = "BASE64"; + +/** File formats Skyflow Detect accepts and produces. */ +export enum FileDataFormat { + BMP = "BMP", + CSV = "CSV", + DCM = "DCM", + DOC = "DOC", + DOCX = "DOCX", + GIF = "GIF", + JPEG = "JPEG", + JPG = "JPG", + JSON = "JSON", + JSONL = "JSONL", + MP3 = "MP3", + PDF = "PDF", + PNG = "PNG", + PPT = "PPT", + PPTX = "PPTX", + TIF = "TIF", + TIFF = "TIFF", + TXT = "TXT", + WAV = "WAV", + XLS = "XLS", + XLSX = "XLSX", + XML = "XML", + ZIP = "ZIP", +} + +/** Status of a Detect file run. `SUCCESS`, `FAILED` and `UNKNOWN` are terminal. */ +export enum DetectRunStatus { + UNKNOWN = "UNKNOWN", + QUEUED = "QUEUED", + IN_PROGRESS = "IN_PROGRESS", + SUCCESS = "SUCCESS", + FAILED = "FAILED", +} + +/** Whether raw entity values are returned on a de-identify response. */ +export enum ReturnEntitiesType { + EXCLUDE_SENSITIVE_DATA = "EXCLUDE_SENSITIVE_DATA", + ALL = "ALL", + NONE = "NONE", +} + +/** How detected regions of an image are masked. */ +export enum MaskingMethod { + NONE = "NONE", + BLUR = "BLUR", + BLACKBOX = "BLACKBOX", +} + +/** Transcription produced for an audio file. */ +export enum OutputTranscriptionType { + NONE = "NONE", + TRANSCRIPTION = "TRANSCRIPTION", + MEDICAL_TRANSCRIPTION = "MEDICAL_TRANSCRIPTION", + DIARIZED_TRANSCRIPTION = "DIARIZED_TRANSCRIPTION", + MEDICAL_DIARIZED_TRANSCRIPTION = "MEDICAL_DIARIZED_TRANSCRIPTION", +} + +/** How a PDF is read before detection. */ +export enum PdfProcessingMode { + NONE = "NONE", + OCR = "OCR", + TEXT_LAYER = "TEXT_LAYER", +} + +/** Visual objects Skyflow Detect can find in images. */ +export enum ObjectEntityType { + NONE = "NONE", + FACE = "FACE", + LICENSE_PLATE = "LICENSE_PLATE", + LOGO = "LOGO", + SIGNATURE = "SIGNATURE", + ALL = "ALL", +} + +/** How a detected visual object is handled. */ +export enum ObjectDeidentificationType { + UNKNOWN = "UNKNOWN", + REDACT = "REDACT", + UNREDACT = "UNREDACT", +} + +/** Overall outcome of a `checkGuardrails` call. */ +export enum GuardrailsValidation { + PASSED = "PASSED", + FAILED = "FAILED", + UNKNOWN = "UNKNOWN", +} + +// --- Inline Detect configuration -------------------------------------------- + +/** Shift detected dates by a random number of days within the range. */ +export interface ShiftDates { + minDays?: number; + maxDays?: number; +} + +/** Transformations applied to a detected entity before it is replaced. */ +export interface Transformation { + shiftDates?: ShiftDates; +} + +/** One entity type to detect and how to de-identify it. */ +export interface Entity { + /** Defaults to `ALL` on the server. */ + entityType?: EntityType; + /** Defaults to `ENTITY_UNIQUE_COUNTER` on the server. */ + deidentificationType?: DeidentificationType; + /** + * Used only with `VAULT_TOKEN`: where the value is stored. A `table.column` + * on skyvault vaults, a token group name on flowvault vaults. Must exist. + */ + destination?: string; + transformation?: Transformation; +} + +/** A regex-defined entity type. `label` is reported as the entity type of its matches. */ +export interface CustomType { + label?: string; + match?: string[]; +} + +/** A visual object type to detect in images and how to handle it. */ +export interface ObjectEntity { + entityType?: ObjectEntityType; + deidentificationType?: ObjectDeidentificationType; +} + +/** What to detect and how to de-identify it. Applies to string and file inputs. */ +export interface Detect { + entities?: Entity[]; + /** File inputs only. */ + objectEntities?: ObjectEntity[]; + /** Regexes whose matches are always treated as sensitive. */ + restrict?: string[]; + /** Regexes whose matches are never treated as sensitive. */ + skip?: string[]; + customTypes?: CustomType[]; + /** Defaults to `EXCLUDE_SENSITIVE_DATA` on the server. */ + returnEntities?: ReturnEntitiesType; +} + +/** Per-column handling for structured files. File inputs only. */ +export interface FileMapping { + source?: string; + destination?: string; + outputFileNameSuffix?: string; + entitiesDestination?: string; + objectEntitiesDestination?: string; +} + +/** The tone laid over detected speech. Paddings are in seconds, frequency in Hz. */ +export interface Bleep { + startPadding?: number; + stopPadding?: number; + frequency?: number; + gain?: number; +} + +/** Audio options for file inputs. */ +export interface Audio { + outputProcessedAudio?: boolean; + outputTranscription?: OutputTranscriptionType; + bleep?: Bleep; +} + +/** PDF options for file inputs. `density` is the rendering DPI. */ +export interface Pdf { + processingMode?: PdfProcessingMode; + density?: number; + maxResolution?: number; +} + +/** Document options for file inputs. */ +export interface Document { + pdf?: Pdf; +} + +/** Image options for file inputs. */ +export interface Image { + outputProcessedImage?: boolean; + outputOcrText?: boolean; + maskingMethod?: MaskingMethod; +} + +/** Media-type options. File inputs only. */ +export interface Media { + audio?: Audio; + document?: Document; + image?: Image; +} + +/** + * The inline form of a stored Detect configuration. `detect` applies to string + * and file inputs; `fileMapping` and `media` apply to file inputs only. The SDK + * fills in the vault id from the controller. + */ +export interface DetectConfiguration { + name?: string; + description?: string; + detect?: Detect; + fileMapping?: FileMapping[]; + media?: Media; +} + +// --- Re-identification ------------------------------------------------------- + +/** + * Contract for one redaction rule of `reidentifyString` and `reidentifyFile`. Each SDK + * exposes its own `RedactionLevel` with the pair its vault understands: `entityName` plus + * `redactionType` on skyvault, `tokenGroupName` plus `redactionPattern` on flowvault. Use + * that type; the fields here are the union the API accepts, and the server rejects + * combinations its vault does not support. + */ +export interface RedactionRule { + /** Skyvault vaults: entity type the rule applies to. */ + entityName?: EntityType; + /** Skyvault vaults: how the value is rendered. */ + redactionType?: RedactionType; + /** Flowvault vaults: token group the rule applies to. */ + tokenGroupName?: string; + /** Flowvault vaults: name of a redaction pattern configured in the vault. */ + redactionPattern?: string; +} + +// --- Polling ----------------------------------------------------------------- + +/** + * Opts `deidentifyFile` into submit-and-poll mode. The SDK polls with + * exponential backoff (1, 2, 4, 8, 16, then 32 seconds between lookups). + */ +export interface PollOptions { + /** Maximum total seconds to keep polling, 1 to 300. Default 60. */ + waitTime?: number; + /** Maximum number of run lookups, 1 to 23. Default 23. */ + maxAttempts?: number; +} + +/** Limits and defaults for {@link PollOptions}. */ +export const POLL_OPTIONS = { + DEFAULT_WAIT_TIME_SECONDS: 60, + MAX_WAIT_TIME_SECONDS: 300, + DEFAULT_MAX_ATTEMPTS: 23, + MAX_MAX_ATTEMPTS: 23, + /** Longest single pause between two lookups, in seconds. */ + MAX_DELAY_SECONDS: 32, +} as const; + +// --- Response components ----------------------------------------------------- + +/** Offsets of a detected entity in the original and processed text. */ +export interface EntityLocation { + startIndex: number | null; + endIndex: number | null; + startIndexProcessed: number | null; + endIndexProcessed: number | null; +} + +/** One entity found by `deidentifyString`. */ +export interface DetectedEntity { + /** The token that replaced the value, for example `NAME_1`. */ + token: string | null; + /** The original value; empty unless `returnEntities` is `ALL`. */ + value: string | null; + location: EntityLocation | null; + /** Entity type label, including custom type labels. */ + entityType: string | null; + /** Confidence scores per entity type. */ + entityScores: Record; +} + +/** Size and count metrics reported for a request. */ +export interface Metrics { + /** Size in kilobytes. */ + size: number | null; + wordCount: number | null; + characterCount: number | null; + /** Presentations only. */ + slides: number | null; + /** Documents only. */ + pages: number | null; + /** Audio only, in seconds. */ + duration: number | null; +} + +/** One output of a file run. */ +export interface FileOutput { + /** Base64 content or a presigned URL, per the response's `outputType`. */ + processedFile: string | null; + /** `REDACTED_FILE`, `ENTITIES`, `OBJECT_ENTITIES`, `REDACTED_TEXT`, and so on. */ + processedFileType: string | null; + processedFileExtension: FileDataFormat | null; +} diff --git a/common/src/error/codes/index.ts b/common/src/error/codes/index.ts index 141bafd9..83be7540 100644 --- a/common/src/error/codes/index.ts +++ b/common/src/error/codes/index.ts @@ -193,45 +193,42 @@ const _SKYFLOW_ERROR_CODE = { EMPTY_VAULT_CLIENTS: { httpCode: 400, message: errorMessages.EMPTY_VAULT_CLIENTS }, EMPTY_CONNECTION_CLIENTS: { httpCode: 400, message: errorMessages.EMPTY_CONNECTION_CLIENTS }, - INVALID_TEXT_IN_DEIDENTIFY: { httpCode: 400, message: errorMessages.INVALID_TEXT_IN_DEIDENTIFY }, - INVALID_ENTITIES_IN_DEIDENTIFY: { httpCode: 400, message: errorMessages.INVALID_ENTITIES_IN_DEIDENTIFY }, - INVALID_ALLOW_REGEX_LIST: { httpCode: 400, message: errorMessages.INVALID_ALLOW_REGEX_LIST }, - INVALID_RESTRICT_REGEX_LIST: { httpCode: 400, message: errorMessages.INVALID_RESTRICT_REGEX_LIST }, - INVALID_TOKEN_FORMAT: { httpCode: 400, message: errorMessages.INVALID_TOKEN_FORMAT }, - TOKEN_FORMAT_NOT_ALLOWED: { httpCode: 400, message: errorMessages.VAULT_TOKEN_FORMAT_NOT_ALLOWED_FOR_DEIDENTIFY_FILES}, - INVALID_TRANSFORMATIONS: { httpCode: 400, message: errorMessages.INVALID_TRANSFORMATIONS }, - - INVALID_TEXT_IN_REIDENTIFY: { httpCode: 400, message: errorMessages.INVALID_TEXT_IN_REIDENTIFY }, - INVALID_REDACTED_ENTITIES_IN_REIDENTIFY: { httpCode: 400, message: errorMessages.INVALID_REDACTED_ENTITIES_IN_REIDENTIFY }, - INVALID_MASKED_ENTITIES_IN_REIDENTIFY: { httpCode: 400, message: errorMessages.INVALID_MASKED_ENTITIES_IN_REIDENTIFY }, - INVALID_PLAIN_TEXT_ENTITIES_IN_REIDENTIFY: { httpCode: 400, message: errorMessages.INVALID_PLAIN_TEXT_ENTITIES_IN_REIDENTIFY }, - - INVALID_DEIDENTIFY_FILE_REQUEST: { httpCode: 400, message: errorMessages.INVALID_DEIDENTIFY_FILE_REQUEST }, - INVALID_DEIDENTIFY_FILE_INPUT: { httpCode: 400, message: errorMessages.INVALID_DEIDENTIFY_FILE_INPUT }, + // Generic file handling EMPTY_FILE_OBJECT:{ httpCode: 400, message: errorMessages.EMPTY_FILE_OBJECT }, INVALID_FILE_FORMAT: { httpCode: 400, message: errorMessages.INVALID_FILE_FORMAT }, MISSING_FILE_SOURCE: { httpCode: 400, message: errorMessages.MISSING_FILE_SOURCE }, INVALID_BASE64_STRING: { httpCode: 400, message: errorMessages.INVALID_BASE64_STRING }, - INVALID_DEIDENTIFY_FILE_OPTIONS: { httpCode: 400, message: errorMessages.INVALID_DEIDENTIFY_FILE_OPTIONS }, - INVALID_ENTITIES: { httpCode: 400, message: errorMessages.INVALID_ENTITIES }, - INVALID_OUTPUT_PROCESSED_IMAGE: { httpCode: 400, message: errorMessages.INVALID_OUTPUT_PROCESSED_IMAGE }, - INVALID_OUTPUT_OCR_TEXT: { httpCode: 400, message: errorMessages.INVALID_OUTPUT_OCR_TEXT }, - INVALID_MASKING_METHOD: { httpCode: 400, message: errorMessages.INVALID_MASKING_METHOD }, - INVALID_PIXEL_DENSITY: { httpCode: 400, message: errorMessages.INVALID_PIXEL_DENSITY }, - INVALID_MAX_RESOLUTION: { httpCode: 400, message: errorMessages.INVALID_MAX_RESOLUTION }, - INVALID_OUTPUT_PROCESSED_AUDIO: { httpCode: 400, message: errorMessages.INVALID_OUTPUT_PROCESSED_AUDIO }, - INVALID_OUTPUT_TRANSCRIPTION: { httpCode: 400, message: errorMessages.INVALID_OUTPUT_TRANSCRIPTION }, - INVALID_BLEEP:{ httpCode: 400, message: errorMessages.INVALID_BLEEP }, - INVALID_FILE_OR_ENCODED_FILE:{ httpCode: 400, message: errorMessages.INVALID_FILE_OR_ENCODED_FILE }, INVALID_FILE_TYPE:{ httpCode: 400, message: errorMessages.INVALID_FILE_TYPE }, - INVALID_DEIDENTIFY_FILE_PATH:{ httpCode: 400, message: errorMessages.INVALID_DEIDENTIFY_FILE_PATH }, FILE_READ_ERROR:{ httpCode: 400, message: errorMessages.FILE_READ_ERROR }, INVALID_BASE64_HEADER:{ httpCode: 400, message: errorMessages.INVALID_BASE64_HEADER }, - INVALID_WAIT_TIME:{ httpCode: 400, message: errorMessages.INVALID_WAIT_TIME }, - INVALID_OUTPUT_DIRECTORY:{ httpCode: 400, message: errorMessages.INVALID_OUTPUT_DIRECTORY }, - INVALID_OUTPUT_DIRECTORY_PATH:{ httpCode: 400, message: errorMessages.INVALID_OUTPUT_DIRECTORY_PATH }, - EMPTY_RUN_ID:{ httpCode: 400, message: errorMessages.EMPTY_RUN_ID }, - INVALID_RUN_ID:{ httpCode: 400, message: errorMessages.INVALID_RUN_ID }, + + // Detect V2 + INVALID_DEIDENTIFY_STRING_REQUEST: { httpCode: 400, message: errorMessages.INVALID_DEIDENTIFY_STRING_REQUEST }, + INVALID_TEXT_IN_DEIDENTIFY_STRING: { httpCode: 400, message: errorMessages.INVALID_TEXT_IN_DEIDENTIFY_STRING }, + INVALID_REIDENTIFY_STRING_REQUEST: { httpCode: 400, message: errorMessages.INVALID_REIDENTIFY_STRING_REQUEST }, + INVALID_TEXT_IN_REIDENTIFY_STRING: { httpCode: 400, message: errorMessages.INVALID_TEXT_IN_REIDENTIFY_STRING }, + INVALID_REDACTION_LEVEL: { httpCode: 400, message: errorMessages.INVALID_REDACTION_LEVEL }, + INVALID_DEIDENTIFY_FILE_REQUEST: { httpCode: 400, message: errorMessages.INVALID_DEIDENTIFY_FILE_REQUEST }, + INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE: { httpCode: 400, message: errorMessages.INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE }, + INVALID_VALUE_IN_DEIDENTIFY_FILE: { httpCode: 400, message: errorMessages.INVALID_VALUE_IN_DEIDENTIFY_FILE }, + MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE: { httpCode: 400, message: errorMessages.MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE }, + MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE: { httpCode: 400, message: errorMessages.MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE }, + INVALID_FILE_IN_DEIDENTIFY_FILE: { httpCode: 400, message: errorMessages.INVALID_FILE_IN_DEIDENTIFY_FILE }, + FILE_NOT_FOUND_IN_DEIDENTIFY_FILE: { httpCode: 400, message: errorMessages.FILE_NOT_FOUND_IN_DEIDENTIFY_FILE }, + FAILED_TO_ENCODE_FILE: { httpCode: 400, message: errorMessages.FAILED_TO_ENCODE_FILE }, + INVALID_WAIT_TIME_IN_POLL_OPTIONS: { httpCode: 400, message: errorMessages.INVALID_WAIT_TIME_IN_POLL_OPTIONS }, + INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS: { httpCode: 400, message: errorMessages.INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS }, + OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS: { httpCode: 400, message: errorMessages.OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS }, + FAILED_TO_WRITE_OUTPUT_FILE: { httpCode: 400, message: errorMessages.FAILED_TO_WRITE_OUTPUT_FILE }, + INVALID_REIDENTIFY_FILE_REQUEST: { httpCode: 400, message: errorMessages.INVALID_REIDENTIFY_FILE_REQUEST }, + INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE: { httpCode: 400, message: errorMessages.INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE }, + INVALID_VALUE_IN_REIDENTIFY_FILE: { httpCode: 400, message: errorMessages.INVALID_VALUE_IN_REIDENTIFY_FILE }, + INVALID_GET_RUN_REQUEST: { httpCode: 400, message: errorMessages.INVALID_GET_RUN_REQUEST }, + INVALID_RUN_ID_IN_GET_RUN: { httpCode: 400, message: errorMessages.INVALID_RUN_ID_IN_GET_RUN }, + INVALID_CHECK_GUARDRAILS_REQUEST: { httpCode: 400, message: errorMessages.INVALID_CHECK_GUARDRAILS_REQUEST }, + INVALID_TEXT_IN_CHECK_GUARDRAILS: { httpCode: 400, message: errorMessages.INVALID_TEXT_IN_CHECK_GUARDRAILS }, + INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS: { httpCode: 400, message: errorMessages.INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS }, + INTERNAL_SERVER_ERROR: { httpCode: 500, message: errorMessages.INTERNAL_SERVER_ERROR }, INVALID_XML_FORMAT: { httpCode: 400, message: errorMessages.INVALID_XML_FORMAT }, }; diff --git a/common/src/error/messages/index.ts b/common/src/error/messages/index.ts index b5c193ca..649d4e38 100644 --- a/common/src/error/messages/index.ts +++ b/common/src/error/messages/index.ts @@ -199,52 +199,44 @@ const errorMessages = { get EMPTY_CONNECTION_CLIENTS(): string { return `${errorPrefix()} Validation error. No connection config found. Please add a connection config`; }, get GENERIC_API_ERROR(): string { return `${errorPrefix()} Validation error. Invalid configuration. Please add a valid vault configuration`; }, - get INVALID_TEXT_IN_DEIDENTIFY(): string { return `${errorPrefix()} Validation error. The text field is required and must be a non-empty string. Specify a valid text.`; }, - get INVALID_ENTITIES_IN_DEIDENTIFY(): string { return `${errorPrefix()} Validation error. The entities field must be an array of DetectEntities enums. Specify a valid entities.`; }, - get INVALID_ALLOW_REGEX_LIST(): string { return `${errorPrefix()} Validation error. The allowRegexList field must be an array of strings. Specify a valid allowRegexList.`; }, - get INVALID_RESTRICT_REGEX_LIST(): string { return `${errorPrefix()} Validation error. The restrictRegexList field must be an array of strings. Specify a valid restrictRegexList.`; }, - get INVALID_TOKEN_FORMAT(): string { return `${errorPrefix()} Validation error. The tokenFormat key must be an instance of TokenFormat. Specify a valid token format.`; }, - get INVALID_TRANSFORMATIONS(): string { return `${errorPrefix()} Validation error. The transformations key must be an instance of Transformations. Specify a valid transformations.`; }, - get VAULT_TOKEN_FORMAT_NOT_ALLOWED_FOR_DEIDENTIFY_FILES(): string { return `${errorPrefix()} Validation error. Vault token format is not allowed for deidentify file request.`; }, - - get INVALID_TEXT_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The text field is required and must be a non-empty string. Specify a valid text.`; }, - get INVALID_REDACTED_ENTITIES_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The redactedEntities field must be an array of DetectEntities enums. Specify a valid redactedEntities.`; }, - get INVALID_MASKED_ENTITIES_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The maskedEntities field must be an array of DetectEntities enums. Specify a valid maskedEntities.`; }, - get INVALID_PLAIN_TEXT_ENTITIES_IN_REIDENTIFY(): string { return `${errorPrefix()} Validation error. The plainTextEntities field must be an array of DetectEntities enums. Specify a valid plainTextEntities.`; }, - - get INVALID_DEIDENTIFY_FILE_REQUEST(): string { return `${errorPrefix()} Validation error. Invalid deidentify file request. Specify a valid deidentify file request.`; }, - get INVALID_DEIDENTIFY_FILE_INPUT(): string { return `${errorPrefix()} Validation error. Invalid deidentify file input. Please provide either a file or a file path.`; }, + // Generic file handling get EMPTY_FILE_OBJECT(): string { return `${errorPrefix()} Validation error. File object cannot be empty. Specify a valid file object.`; }, get INVALID_FILE_FORMAT(): string { return `${errorPrefix()} Validation error. Invalid file format. Specify a valid file format.`; }, get MISSING_FILE_SOURCE(): string { return `${errorPrefix()} Validation error. Provide exactly one of filePath, base64, or fileObject.`; }, get INVALID_FILE_OBJECT(): string { return `${errorPrefix()} Validation error. Invalid file object. Specify a valid file object.`; }, get INVALID_BASE64_STRING(): string { return `${errorPrefix()} Validation error. Invalid base64 string. Specify a valid base64 string.`; }, - get INVALID_DEIDENTIFY_FILE_OPTIONS(): string { return `${errorPrefix()} Validation error. Invalid deidentify file options. Specify a valid deidentify file options.`; }, - get INVALID_ENTITIES(): string { return `${errorPrefix()} Validation error. Invalid entities. Specify valid entities as string array.`; }, - get EMPTY_ENTITIES(): string { return `${errorPrefix()} Validation error. Entities cannot be empty. Specify valid entities.`; }, - get EMPTY_ALLOW_REGEX_LIST(): string { return `${errorPrefix()} Validation error. Allow regex list cannot be empty. Specify valid allow regex list.`; }, - get INVALID_ALLOW_REGEX(): string { return `${errorPrefix()} Validation error. Invalid allow regex. Specify valid allow regex at index %s1.`; }, - get EMPTY_RESTRICT_REGEX_LIST(): string { return `${errorPrefix()} Validation error. Restrict regex list cannot be empty. Specify valid restrict regex list.`; }, - get INVALID_RESTRICT_REGEX(): string { return `${errorPrefix()} Validation error. Invalid restrict regex. Specify valid restrict regex at index %s1.`; }, - get INVALID_OUTPUT_PROCESSED_IMAGE(): string { return `${errorPrefix()} Validation error. Invalid output processed image. Specify valid output processed image as string.`; }, - get INVALID_OUTPUT_OCR_TEXT(): string { return `${errorPrefix()} Validation error. Invalid output ocr text. Specify valid output ocr text as string.`; }, - get INVALID_MASKING_METHOD(): string { return `${errorPrefix()} Validation error. Invalid masking method. Specify valid masking method as string.`; }, - get INVALID_PIXEL_DENSITY(): string { return `${errorPrefix()} Validation error. Invalid pixel density. Specify valid pixel density as string.`; }, - get INVALID_OUTPUT_TRANSCRIPTION(): string { return `${errorPrefix()} Validation error. Invalid output transcription. Specify valid output transcription as string.`; }, - get INVALID_OUTPUT_PROCESSED_AUDIO(): string { return `${errorPrefix()} Validation error. Invalid output processed audio. Specify valid output processed audio as string.`; }, - get INVALID_MAX_RESOLUTION(): string { return `${errorPrefix()} Validation error. Invalid max resolution. Specify valid max resolution as string.`; }, - get INVALID_BLEEP(): string { return `${errorPrefix()} Validation error. Invalid bleep. Specify valid bleep as object.`; }, - get INVALID_FILE_OR_ENCODED_FILE(): string { return `${errorPrefix()} . Error while decoding base64 and saving file`; }, get INVALID_FILE_TYPE(): string { return `${errorPrefix()} Validation error. Invalid file type. Specify a valid file type.`; }, - get INVALID_DEIDENTIFY_FILE_PATH(): string { return `${errorPrefix()} Validation error. Invalid file path. Specify a valid file path.`; }, get INVALID_FILE_NAME(): string { return `${errorPrefix()} Validation error. Invalid file name. Specify a valid file name.`; }, get FILE_READ_ERROR(): string { return `${errorPrefix()} Validation error. Unable to read file. Verify the file path.`; }, get INVALID_BASE64_HEADER(): string { return `${errorPrefix()} Validation error. Invalid base64 header. Specify a valid base64 header.`; }, - get INVALID_WAIT_TIME(): string { return `${errorPrefix()} Validation error. Invalid wait time. Specify a valid wait time as number and should not be greater than 20 secs.`; }, - get INVALID_OUTPUT_DIRECTORY(): string { return `${errorPrefix()} Validation error. Invalid output directory. Specify a valid output directory as string.`; }, - get INVALID_OUTPUT_DIRECTORY_PATH(): string { return `${errorPrefix()} Validation error. Invalid output directory path. Specify a valid output directory path as string.`; }, - get EMPTY_RUN_ID(): string { return `${errorPrefix()} Validation error. Run id cannot be empty. Specify a valid run id.`; }, - get INVALID_RUN_ID(): string { return `${errorPrefix()} Validation error. Invalid run id. Specify a valid run id as string.`; }, + + // Detect V2 + get INVALID_DEIDENTIFY_STRING_REQUEST(): string { return `${errorPrefix()} Validation error. DeidentifyString request cannot be null.`; }, + get INVALID_TEXT_IN_DEIDENTIFY_STRING(): string { return `${errorPrefix()} Validation error. Invalid text in deidentify string request. Specify a non-empty text.`; }, + get INVALID_REIDENTIFY_STRING_REQUEST(): string { return `${errorPrefix()} Validation error. ReidentifyString request cannot be null.`; }, + get INVALID_TEXT_IN_REIDENTIFY_STRING(): string { return `${errorPrefix()} Validation error. Invalid text in reidentify string request. Specify a non-empty text.`; }, + get INVALID_REDACTION_LEVEL(): string { return `${errorPrefix()} Validation error. redactionLevel must be an array of RedactionLevel objects.`; }, + get INVALID_DEIDENTIFY_FILE_REQUEST(): string { return `${errorPrefix()} Validation error. DeidentifyFile request cannot be null.`; }, + get INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. Invalid dataSource in deidentify file request. Specify a DeidentifySourceType.`; }, + get INVALID_VALUE_IN_DEIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. Invalid value in deidentify file request. Specify the non-empty content or reference that dataSource describes.`; }, + get MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. Specify only one of filePath, file or value in deidentify file request.`; }, + get MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. Specify one of filePath, file or value in deidentify file request.`; }, + get INVALID_FILE_IN_DEIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. Invalid file in deidentify file request. Specify a File object.`; }, + get FILE_NOT_FOUND_IN_DEIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. The file to deidentify was not found at the specified path. Verify the file path and try again.`; }, + get FAILED_TO_ENCODE_FILE(): string { return `${errorPrefix()} Validation error. Failed to encode the file. Ensure the file is in a supported format and try again.`; }, + get INVALID_WAIT_TIME_IN_POLL_OPTIONS(): string { return `${errorPrefix()} Validation error. waitTime in pollOptions must be between 1 and 300 seconds.`; }, + get INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS(): string { return `${errorPrefix()} Validation error. maxAttempts in pollOptions must be between 1 and 23.`; }, + get OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS(): string { return `${errorPrefix()} Validation error. outputDirectory in deidentify file request requires pollOptions.`; }, + get FAILED_TO_WRITE_OUTPUT_FILE(): string { return `${errorPrefix()} Failed to write the processed output to %s1.`; }, + get INVALID_REIDENTIFY_FILE_REQUEST(): string { return `${errorPrefix()} Validation error. ReidentifyFile request cannot be null.`; }, + get INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. Invalid dataSource in reidentify file request. Specify BASE64 or PRESIGNED_URL.`; }, + get INVALID_VALUE_IN_REIDENTIFY_FILE(): string { return `${errorPrefix()} Validation error. Invalid value in reidentify file request. Specify the non-empty base64 content or presigned URL of the file.`; }, + get INVALID_GET_RUN_REQUEST(): string { return `${errorPrefix()} Validation error. GetRun request cannot be null.`; }, + get INVALID_RUN_ID_IN_GET_RUN(): string { return `${errorPrefix()} Validation error. Invalid runId in get run request. Specify a non-empty run id.`; }, + get INVALID_CHECK_GUARDRAILS_REQUEST(): string { return `${errorPrefix()} Validation error. CheckGuardrails request cannot be null.`; }, + get INVALID_TEXT_IN_CHECK_GUARDRAILS(): string { return `${errorPrefix()} Validation error. Invalid text in check guardrails request. Specify a non-empty text.`; }, + get INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS(): string { return `${errorPrefix()} Validation error. denyTopics in check guardrails request must be an array of strings.`; }, + get INTERNAL_SERVER_ERROR(): string { return `${errorPrefix()}. Internal server error. %s1.`; }, get INVALID_XML_FORMAT(): string { return `${errorPrefix()} Validation error. Invalid XML format. Specify a valid XML format as string.`; }, }; diff --git a/common/src/index.ts b/common/src/index.ts index 7f8065e3..982d90cf 100644 --- a/common/src/index.ts +++ b/common/src/index.ts @@ -5,8 +5,7 @@ // --- Utils: enums, consts, helpers and shared interfaces --------------------- // Exposes Env, LogLevel, RequestMethod, RedactionType, OrderByEnum, TokenMode, -// TokenType, MaskingMethod, DetectOutputTranscription, DetectEntities, MessageType, -// AuthType, CONFIG, CONTROLLER_TYPES, TYPES, SKYFLOW, SDK, HTTP_STATUS_CODE, +// MessageType, AuthType, CONFIG, CONTROLLER_TYPES, TYPES, SKYFLOW, SDK, HTTP_STATUS_CODE, // CONTENT_TYPE, HTTP_HEADER, REQUEST (and other consts); printLog, parameterizedString, // generateSDKMetrics, getVaultURL, getConnectionBaseURL, getBearerToken, getToken, // validateToken, removeSDKVersion, getBaseUrl, fillUrlWithPathAndQueryParams, @@ -34,9 +33,8 @@ export { default as VaultConfig } from "./config/vault"; export { default as ConnectionConfig } from "./config/connection"; // --- Shared, contract-agnostic types ----------------------------------------- -// SkyflowConfig, StringKeyValueMapType, FileInput, Filepath, FileObject, -// InsertResponseType, GetResponseData, QueryResponseType, TokenizeRequestType, -// IndexRange, ServiceAccountResponseError. +// SkyflowConfig, StringKeyValueMapType, InsertResponseType, GetResponseData, +// QueryResponseType, TokenizeRequestType, ServiceAccountResponseError. export * from "./types"; // --- Service account (token generation) -------------------------------------- @@ -80,3 +78,8 @@ export * from "./base/response"; // --- Facade base (shared Skyflow client scaffolding) ------------------------- export { BaseSkyflow } from "./skyflow/base"; export type { BaseVaultEntry, BaseVaultClientLike } from "./skyflow/base"; + +// --- Detect V2 (shared by skyvault and flowvault) ---------------------------- +// Request / response models, inline-configuration types, enums, and the +// BaseDetectController each layer's DetectController extends. +export * from "./detect"; diff --git a/common/src/service-account/index.ts b/common/src/service-account/index.ts index 320884b8..b406866b 100644 --- a/common/src/service-account/index.ts +++ b/common/src/service-account/index.ts @@ -1,6 +1,6 @@ import * as fs from 'fs'; import jwt from "jsonwebtoken"; -import { V1GetAuthTokenRequest, V1GetAuthTokenResponse } from '../_generated_/rest/api'; +import { GetAuthTokenRequest, GetAuthTokenResponse } from '../_generated_/rest/api'; import { getBaseUrl, LogLevel, isValidURL, MessageType, parameterizedString, printLog, HTTP_HEADER, CONTENT_TYPE, HTTP_STATUS_CODE, JWT, ENCODING_TYPE } from '../utils'; import Client from './client'; import logs from '../utils/logs'; @@ -165,15 +165,15 @@ function getToken(credentials, options?: BearerTokenOptions): Promise) => { + ).withRawResponse().then((res: WithRawResponse) => { successResponse(res.data, options?.logLevel).then((response) => resolve(response)).catch(err => reject(err)) }) .catch((err) => { @@ -356,7 +356,7 @@ function failureResponse(err: ServiceAccountResponseError, options?: BearerToken }) } -function successResponse(res: V1GetAuthTokenResponse, logLevel?: LogLevel): Promise { +function successResponse(res: GetAuthTokenResponse, logLevel?: LogLevel): Promise { printLog(logs.infoLogs.GENERATE_BEARER_TOKEN_SUCCESS, MessageType.LOG, logLevel); return Promise.resolve({ accessToken: res.accessToken ?? '', diff --git a/common/src/types/index.ts b/common/src/types/index.ts index 1f381db2..ad1004fc 100644 --- a/common/src/types/index.ts +++ b/common/src/types/index.ts @@ -10,18 +10,6 @@ export interface SkyflowConfig { logLevel?: LogLevel; } -export type FileInput = - | Filepath - | FileObject - -export interface Filepath { - filePath: string; -} - -export interface FileObject { - file: File; -} - export interface InsertResponseType { skyflowId: string; /** @deprecated Renamed to skyflowId. Will be removed in v3. */ @@ -46,11 +34,6 @@ export interface TokenizeRequestType { value: string; } -export interface IndexRange { - start?: number; - end?: number; -} - export type ServiceAccountResponseError = { rawResponse?: { headers?: { diff --git a/common/src/utils/index.ts b/common/src/utils/index.ts index 606f3575..eb7dd788 100644 --- a/common/src/utils/index.ts +++ b/common/src/utils/index.ts @@ -103,24 +103,8 @@ export const TYPES = { QUERY: 'QUERY', DETECT: 'DETECT', INVOKE_CONNECTION: 'INVOKE_CONNECTION', - DEIDENTIFY_TEXT: 'DEIDENTIFY_TEXT', - REIDENTIFY_TEXT: 'REIDENTIFY_TEXT', - DEIDENTIFY_FILE: 'DEIDENTIFY_FILE', - DETECT_RUN: 'DETECT_RUN', }; -export enum DeidenitfyFileRequestTypes { - IMAGE= "IMAGE", - FILE= 'FILE', - AUDIO= 'AUDIO', - PPT= 'PPT', - DOCUMENT= 'DOCUMENT', - PDF= 'PDF', - SPREADSHEET= 'SPREADSHEET', - STRUCTURED_TEXT= 'STRUCTURED_TEXT', - TEXT='TEXT' -} - export const CONTROLLER_TYPES = { DETECT: 'DETECT', VAULT: 'VAULT', @@ -128,98 +112,6 @@ export const CONTROLLER_TYPES = { } -export enum DetectOutputTranscription { - DIARIZED_TRANSCRIPTION = "diarized_transcription", - MEDICAL_DIARIZED_TRANSCRIPTION = "medical_diarized_transcription", - MEDICAL_TRANSCRIPTION = "medical_transcription", - PLAINTEXT_TRANSCRIPTION = "plaintext_transcription", - TRANSCRIPTION = "transcription", -} - -export enum MaskingMethod{ - Blackbox= "blackbox", - Blur= "blur", -} - -export enum DetectEntities { - ACCOUNT_NUMBER = "account_number", - AGE = 'age', - ALL = 'all', - BANK_ACCOUNT = 'bank_account', - BLOOD_TYPE = 'blood_type', - CONDITION = 'condition', - CORPORATE_ACTION = 'corporate_action', - CREDIT_CARD = 'credit_card', - CREDIT_CARD_EXPIRATION = 'credit_card_expiration', - CVV = 'cvv', - DAY = 'day', - DATE = 'date', - DATE_INTERVAL = 'date_interval', - DOB = 'dob', - DOSE = 'dose', - DRIVER_LICENSE = 'driver_license', - DRUG = 'drug', - DURATION = 'duration', - EFFECT = 'effect', - EMAIL_ADDRESS = 'email_address', - EVENT = 'event', - FILENAME = 'filename', - FINANCIAL_METRIC = 'financial_metric', - GENDER = 'gender', - HEALTHCARE_NUMBER = 'healthcare_number', - INJURY = 'injury', - IP_ADDRESS = 'ip_address', - LANGUAGE = 'language', - LOCATION = 'location', - LOCATION_ADDRESS = 'location_address', - LOCATION_ADDRESS_STREET = 'location_address_street', - LOCATION_CITY = 'location_city', - LOCATION_COORDINATE = 'location_coordinate', - LOCATION_COUNTRY = 'location_country', - LOCATION_STATE = 'location_state', - LOCATION_ZIP = 'location_zip', - MARITAL_STATUS = 'marital_status', - MEDICAL_CODE = 'medical_code', - MEDICAL_PROCESS = 'medical_process', - MONEY = 'money', - MONTH = 'month', - NAME = 'name', - NAME_FAMILY = 'name_family', - NAME_GIVEN = 'name_given', - NAME_MEDICAL_PROFESSIONAL = 'name_medical_professional', - NUMERICAL_PII = 'numerical_pii', - OCCUPATION = 'occupation', - ORGANIZATION = 'organization', - ORGANIZATION_ID = 'organization_id', - ORGANIZATION_MEDICAL_FACILITY = 'organization_medical_facility', - ORIGIN = 'origin', - PASSPORT_NUMBER = 'passport_number', - PASSWORD = 'password', - PHONE_NUMBER = 'phone_number', - PROJECT = 'project', - PHYSICAL_ATTRIBUTE = 'physical_attribute', - POLITICAL_AFFILIATION = 'political_affiliation', - PRODUCT = 'product', - RELIGION = 'religion', - ROUTING_NUMBER = 'routing_number', - SEXUALITY = 'sexuality', - SSN = 'ssn', - STATISTICS = 'statistics', - TIME = 'time', - TREND = 'trend', - URL = 'url', - USERNAME = 'username', - VEHICLE_ID = 'vehicle_id', - ZODIAC_SIGN = 'zodiac_sign', -} - -export enum TokenType { - ENTITY_UNIQUE_COUNTER = 'entity_unq_counter', - ENTITY_ONLY = 'entity_only', - VAULT_TOKEN = 'vault_token' -} - - // HTTP Status Codes export const HTTP_STATUS_CODE = { OK: 200, @@ -247,33 +139,6 @@ export const HTTP_HEADER = { ERROR_FROM_CLIENT: 'error-from-client', } as const; -// Detect API Status Values -export const DETECT_STATUS = { - IN_PROGRESS: 'IN_PROGRESS', - SUCCESS: 'SUCCESS', - FAILED: 'FAILED', -} as const; - -// File Extensions -export const FILE_EXTENSION = { - JSON: 'json', - MP3: 'mp3', - WAV: 'wav', -} as const; - -// File Format Types -export const FILE_FORMAT_TYPE = { - TXT: 'txt', - PDF: 'pdf', -} as const; - -// File Processing -export const FILE_PROCESSING = { - PROCESSED_PREFIX: 'processed-', - DEIDENTIFIED_PREFIX: 'deidentified.', - ENTITIES: 'entities', -} as const; - // Encoding Types export const ENCODING_TYPE = { UTF8: 'utf8', diff --git a/common/src/utils/logs/index.ts b/common/src/utils/logs/index.ts index ce8b0084..9bb2a4cc 100644 --- a/common/src/utils/logs/index.ts +++ b/common/src/utils/logs/index.ts @@ -61,25 +61,43 @@ const logs = { GENERATE_SIGNED_DATA_TOKENS_TRIGGERED: "generateSignedDataTokens is triggered", UPDATE_TRIGGERED: 'Update method triggered.', UPDATE_REQUEST_RESOLVED: 'Update request is resolved.', - DETECT_REQUEST_RESOLVED: 'Detect request is resolved.', - DEIDENTIFY_FILE_REQUEST_RESOLVED: 'Deidentify file request is resolved.', - DETECT_RUN_REQUEST_RESOLVED: 'Detect get run request is resolved.', - DEIDENTIFY_TEXT_REQUEST_RESOLVED: 'Deidentify text request is resolved.', - REIDENTIFY_TEXT_REQUEST_RESOLVED: 'Reidentify text request is resolved.', UNABLE_TO_GENERATE_SDK_METRIC: 'Unable to generate %s1 metric.', USING_BEARER_TOKEN: 'Using token from credentials', USING_KEY: 'Using api key from credentials', USING_CREDENTIALS_STRING: 'Using credentials string from credentials', USING_PATH: 'Using path from credentials', USING_SKYFLOW_CREDENTIALS_ENV: 'Using SKYFLOW_CREDENTIALS from env', - DEIDENTIFY_TEXT_TRIGGERED: 'Deidentify text method triggered', - VALIDATE_DEIDENTIFY_TEXT_INPUT: 'Validating deidentify text input', - REIDENTIFY_TEXT_TRIGGERED: 'Reidentify text method triggered', - VALIDATE_REIDENTIFY_TEXT_INPUT: 'Validating reidentify text input', - VALIDATE_DETECT_FILE_INPUT: 'Validating deidentify file input.', - DETECT_FILE_TRIGGERED: 'Deidentify file triggered.', - GET_DETECT_RUN_TRIGGERED: 'Get detect run triggered.', - VALIDATE_GET_DETECT_RUN_INPUT: 'Validating get detect run input.', + // Detect V2 + DEIDENTIFY_STRING_TRIGGERED: 'DeIdentify string method triggered.', + VALIDATE_DEIDENTIFY_STRING_REQUEST: 'Validating deidentify string request.', + DEIDENTIFY_STRING_REQUEST_RESOLVED: 'DeIdentify string request resolved.', + DEIDENTIFY_STRING_SUCCESS: 'String data de-identified.', + REIDENTIFY_STRING_TRIGGERED: 'ReIdentify string method triggered.', + VALIDATE_REIDENTIFY_STRING_REQUEST: 'Validating reidentify string request.', + REIDENTIFY_STRING_REQUEST_RESOLVED: 'ReIdentify string request resolved.', + REIDENTIFY_STRING_SUCCESS: 'String data re-identified.', + DEIDENTIFY_FILE_TRIGGERED: 'DeIdentify file method triggered.', + VALIDATE_DEIDENTIFY_FILE_REQUEST: 'Validating deidentify file request.', + DEIDENTIFY_FILE_REQUEST_RESOLVED: 'DeIdentify file request resolved.', + DEIDENTIFY_FILE_RUN_SUBMITTED: 'DeIdentify file run %s1 submitted; retrieve it with getRun.', + DEIDENTIFY_FILE_SUCCESS: 'File data de-identified.', + REIDENTIFY_FILE_TRIGGERED: 'ReIdentify file method triggered.', + VALIDATE_REIDENTIFY_FILE_REQUEST: 'Validating reidentify file request.', + REIDENTIFY_FILE_REQUEST_RESOLVED: 'ReIdentify file request resolved.', + REIDENTIFY_FILE_SUCCESS: 'File data re-identified.', + GET_RUN_TRIGGERED: 'Get run method triggered.', + VALIDATE_GET_RUN_REQUEST: 'Validating get run request.', + GET_RUN_REQUEST_RESOLVED: 'Get run request resolved.', + GET_RUN_SUCCESS: 'Detect run retrieved.', + CHECK_GUARDRAILS_TRIGGERED: 'Check guardrails method triggered.', + VALIDATE_CHECK_GUARDRAILS_REQUEST: 'Validating check guardrails request.', + CHECK_GUARDRAILS_REQUEST_RESOLVED: 'Check guardrails request resolved.', + CHECK_GUARDRAILS_SUCCESS: 'Guardrails check completed.', + POLL_RUN_STARTED: 'Polling run %s1 for up to %s2 seconds and %s3 attempts.', + POLL_RUN_TERMINAL: 'Run %s1 reached status %s2 after %s3 attempts.', + POLL_RUN_BUDGET_EXHAUSTED: 'Run %s1 still %s2 after %s3 attempts and %s4 seconds; returning run id.', + OUTPUT_FILE_WRITTEN: 'Processed output written to %s1.', + OUTPUT_NOT_WRITTEN_FOR_OUTPUT_TYPE: 'Outputs delivered as %s1 are not written to disk.', }, errorLogs: { VAULT_CONFIG_KEY_MISSING: "Invalid skyflow config. Vaults configs key missing in skyflow config.", @@ -171,14 +189,40 @@ const logs = { INVALID_TABLE_IN_FILE_UPLOAD: "Invalid file upload request. Table name is required.", EMPTY_SKYFLOW_ID_IN_FILE_UPLOAD: "Invalid file upload request. Skyflow Id can not be empty.", INVALID_SKYFLOW_ID_IN_FILE_UPLOAD: "Invalid file upload request. Skyflow Id is required.", - EMPTY_RUN_ID: "Invalid Run Id. Run Id can not be empty.", - INVALID_RUN_ID: "Invalid Run ID. A Run ID of string type is required.", INVALID_TOKEN_URI: "Invalid credentials. Token URI must be a string and a valid URL.", - DETECT_REQUEST_RESOLVED: 'Detect request is resolved.', - DEIDENTIFY_FILE_REQUEST_REJECTED: 'Deidentify file resulted in failure.', - DETECT_RUN_REQUEST_REJECTED: 'Detect get run resulted in failure.', - DEIDENTIFY_TEXT_REQUEST_REJECTED: 'Deidentify text resulted in failure.', - REIDENTIFY_TEXT_REQUEST_REJECTED: 'Reidentify text resulted in failure.', + // Detect V2 (%s1 is the interface name, "detect") + DEIDENTIFY_STRING_REQUEST_NULL: "Invalid %s1 request. DeidentifyString request can not be null.", + INVALID_TEXT_IN_DEIDENTIFY_STRING: "Invalid %s1 request. The text field is required and must be a non-empty string. Specify a valid text.", + DEIDENTIFY_STRING_REQUEST_REJECTED: "DeIdentify string request resulted in failure.", + REIDENTIFY_STRING_REQUEST_NULL: "Invalid %s1 request. ReidentifyString request can not be null.", + INVALID_TEXT_IN_REIDENTIFY_STRING: "Invalid %s1 request. The text field is required and must be a non-empty string. Specify a valid text.", + INVALID_REDACTION_LEVEL: "Invalid %s1 request. The redactionLevel field must be an array of RedactionLevel objects.", + REIDENTIFY_STRING_REQUEST_REJECTED: "ReIdentify string request resulted in failure.", + DEIDENTIFY_FILE_REQUEST_NULL: "Invalid %s1 request. DeidentifyFile request can not be null.", + INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE: "Invalid %s1 request. The dataSource field is required with value. Specify a DeidentifySourceType.", + INVALID_VALUE_IN_DEIDENTIFY_FILE: "Invalid %s1 request. The value field must be a non-empty string.", + MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE: "Invalid %s1 request. Specify only one of filePath, file or value.", + MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE: "Invalid %s1 request. Specify one of filePath, file or value.", + INVALID_FILE_IN_DEIDENTIFY_FILE: "Invalid %s1 request. The file field must be a File object.", + FILE_NOT_FOUND_IN_DEIDENTIFY_FILE: "Invalid %s1 request. The file to deidentify was not found.", + FAILED_TO_ENCODE_FILE_IN_DEIDENTIFY_FILE: "Invalid %s1 request. Failed to read and encode the file to deidentify.", + INVALID_WAIT_TIME_IN_POLL_OPTIONS: "Invalid %s1 request. waitTime in pollOptions must be between 1 and 300.", + INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS: "Invalid %s1 request. maxAttempts in pollOptions must be between 1 and 23.", + OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS: "Invalid %s1 request. outputDirectory requires pollOptions.", + DEIDENTIFY_FILE_REQUEST_REJECTED: "DeIdentify file request resulted in failure.", + OUTPUT_DIRECTORY_NOT_CREATED: "Could not create output directory %s1.", + OUTPUT_FILE_NOT_WRITTEN: "Could not write processed output to %s1.", + REIDENTIFY_FILE_REQUEST_NULL: "Invalid %s1 request. ReidentifyFile request can not be null.", + INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE: "Invalid %s1 request. The dataSource field is required. Specify BASE64 or PRESIGNED_URL.", + INVALID_VALUE_IN_REIDENTIFY_FILE: "Invalid %s1 request. The value field is required and must be a non-empty string. Specify the base64 content or presigned URL of the file.", + REIDENTIFY_FILE_REQUEST_REJECTED: "ReIdentify file request resulted in failure.", + GET_RUN_REQUEST_NULL: "Invalid %s1 request. GetRun request can not be null.", + INVALID_RUN_ID_IN_GET_RUN: "Invalid %s1 request. The runId field is required and must be a non-empty string. Specify a valid run id.", + GET_RUN_REQUEST_REJECTED: "Get run request resulted in failure.", + CHECK_GUARDRAILS_REQUEST_NULL: "Invalid %s1 request. CheckGuardrails request can not be null.", + INVALID_TEXT_IN_CHECK_GUARDRAILS: "Invalid %s1 request. The text field is required and must be a non-empty string. Specify a valid text.", + INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS: "Invalid %s1 request. The denyTopics field must be an array of strings.", + CHECK_GUARDRAILS_REQUEST_REJECTED: "Check guardrails request resulted in failure.", }, warnLogs: { DEPRECATED_FILE_UPLOAD_CONSTRUCTOR: "[DEPRECATED] FileUploadRequest(table, skyflowId, columnName) is deprecated and will be removed in a future release. Use FileUploadRequest(table, columnName) with FileUploadOptions.setSkyflowId(skyflowId) instead.", diff --git a/common/test/detect/controller.test.js b/common/test/detect/controller.test.js new file mode 100644 index 00000000..7bd2492a --- /dev/null +++ b/common/test/detect/controller.test.js @@ -0,0 +1,315 @@ +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; + +// Replace the generated Detect V2 resource clients with stubs that record calls. +jest.mock('../../src/_generated_/rest/api/resources/stringsV2/client/Client', () => { + const api = { deidentifyStringV2: jest.fn(), reidentifyStringV2: jest.fn() }; + return { StringsV2: jest.fn(() => api), __api: api }; +}); +jest.mock('../../src/_generated_/rest/api/resources/filesV2/client/Client', () => { + const api = { deidentifyFileV2: jest.fn(), reidentifyFileV2: jest.fn(), getRunV2: jest.fn() }; + return { FilesV2: jest.fn(() => api), __api: api }; +}); +jest.mock('../../src/_generated_/rest/api/resources/guardrailsV2/client/Client', () => { + const api = { checkGuardrailsV2: jest.fn() }; + return { GuardrailsV2: jest.fn(() => api), __api: api }; +}); + +import { BaseDetectController } from '../../src/detect/controller'; +import { StringsV2, __api as strings } from '../../src/_generated_/rest/api/resources/stringsV2/client/Client'; +import { FilesV2, __api as files } from '../../src/_generated_/rest/api/resources/filesV2/client/Client'; +import { __api as guardrails } from '../../src/_generated_/rest/api/resources/guardrailsV2/client/Client'; +import DeidentifyStringRequest from '../../src/detect/model/request/deidentify-string'; +import DeidentifyFileRequest from '../../src/detect/model/request/deidentify-file'; +import ReidentifyStringRequest from '../../src/detect/model/request/reidentify-string'; +import ReidentifyFileRequest from '../../src/detect/model/request/reidentify-file'; +import GetRunRequest from '../../src/detect/model/request/get-run'; +import CheckGuardrailsRequest from '../../src/detect/model/request/check-guardrails'; +import DeidentifyStringResponse from '../../src/detect/model/response/deidentify-string'; +import DeidentifyFileResponse from '../../src/detect/model/response/deidentify-file'; +import ReidentifyStringResponse from '../../src/detect/model/response/reidentify-string'; +import ReidentifyFileResponse from '../../src/detect/model/response/reidentify-file'; +import GetRunResponse from '../../src/detect/model/response/get-run'; +import CheckGuardrailsResponse from '../../src/detect/model/response/check-guardrails'; +import { DetectRunStatus, GuardrailsValidation } from '../../src/detect/types'; +import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType, TestDetectController } from './test-types'; +import { LogLevel, SDK } from '../../src/utils'; +import SkyflowError from '../../src/error'; +import SKYFLOW_ERROR_CODE from '../../src/error/codes'; + +beforeAll(() => { + jest.spyOn(console, 'error').mockImplementation(() => undefined); + jest.spyOn(console, 'log').mockImplementation(() => undefined); +}); +afterAll(() => jest.restoreAllMocks()); + +const URL = 'https://abc.vault.skyflowapis.com'; + +function fakeClient(overrides = {}) { + return { + vaultId: 'vault123', + url: URL, + getLogLevel: () => LogLevel.ERROR, + resolveAuthToken: jest.fn().mockResolvedValue('tok-1'), + ...overrides, + }; +} + +/** No real waiting while polling. */ +class TestController extends TestDetectController { + sleeper() { + return async () => undefined; + } +} + +const apiError = (statusCode, message, requestId = 'req-1') => ({ + statusCode, + body: { error: { message, grpc_code: 3, http_code: statusCode, http_status: 'Bad Request', details: [] } }, + rawResponse: { headers: new Map([['x-request-id', requestId]]) }, +}); + +beforeEach(() => { + jest.clearAllMocks(); +}); + +describe('construction and transport plumbing', () => { + test('builds each generated client lazily with live baseUrl/token suppliers and per-call options', async () => { + const client = fakeClient({ timeout: 30, maxRetries: 2 }); + const controller = new TestController(client); + expect(StringsV2).not.toHaveBeenCalled(); + + strings.deidentifyStringV2.mockResolvedValue({ processedText: 'x' }); + await controller.deidentifyString(new DeidentifyStringRequest('hi')); + + expect(StringsV2).toHaveBeenCalledTimes(1); + const suppliers = StringsV2.mock.calls[0][0]; + expect(suppliers.baseUrl()).toBe(URL); + expect(suppliers.token()).toBe('tok-1'); + + const [, options] = strings.deidentifyStringV2.mock.calls[0]; + expect(options.timeoutInSeconds).toBe(30); + expect(options.maxRetries).toBe(2); + expect(JSON.parse(options.headers[SDK.METRICS_HEADER_KEY])).toEqual(expect.objectContaining({ sdk_name_version: expect.any(String) })); + + // Second call reuses the client; the token supplier reflects the refreshed token. + client.resolveAuthToken.mockResolvedValue('tok-2'); + await controller.deidentifyString(new DeidentifyStringRequest('again')); + expect(StringsV2).toHaveBeenCalledTimes(1); + expect(suppliers.token()).toBe('tok-2'); + }); + + test('omits timeout and retries from the call options when the client has none', async () => { + const controller = new TestController(fakeClient()); + strings.deidentifyStringV2.mockResolvedValue({}); + await controller.deidentifyString(new DeidentifyStringRequest('hi')); + const [, options] = strings.deidentifyStringV2.mock.calls[0]; + expect(options.timeoutInSeconds).toBeUndefined(); + expect(options.maxRetries).toBeUndefined(); + }); + + test('validation failures throw before any token or network call', async () => { + const client = fakeClient(); + const controller = new TestController(client); + await expect(controller.deidentifyString(new DeidentifyStringRequest(''))).rejects.toBeInstanceOf(SkyflowError); + expect(client.resolveAuthToken).not.toHaveBeenCalled(); + expect(strings.deidentifyStringV2).not.toHaveBeenCalled(); + }); + + test('a token failure propagates as-is', async () => { + const client = fakeClient({ resolveAuthToken: jest.fn().mockRejectedValue(new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_CREDENTIALS)) }); + const controller = new TestController(client); + await expect(controller.checkGuardrails(new CheckGuardrailsRequest('t'))).rejects.toMatchObject({ + message: SKYFLOW_ERROR_CODE.EMPTY_CREDENTIALS.message, + }); + }); +}); + +describe('deidentifyString / reidentifyString', () => { + test('deidentifyString sends the mapped request and returns a mapped response', async () => { + const controller = new TestController(fakeClient()); + strings.deidentifyStringV2.mockResolvedValue({ + processedText: 'Hi [NAME_1]', + entities: [{ token: 'NAME_1', entityType: 'NAME', location: { startIndex: 3, endIndex: 7 } }], + metrics: { wordCount: 2 }, + }); + const res = await controller.deidentifyString(new DeidentifyStringRequest('Hi John', { configurationId: 'cfg' })); + expect(strings.deidentifyStringV2.mock.calls[0][0]).toEqual({ text: 'Hi John', configurationId: 'cfg' }); + expect(res).toBeInstanceOf(DeidentifyStringResponse); + expect(res.processedText).toBe('Hi [NAME_1]'); + expect(res.entities[0].token).toBe('NAME_1'); + expect(res.metrics.wordCount).toBe(2); + }); + + test('an API error is normalized into a SkyflowError with code, message and request id', async () => { + const controller = new TestController(fakeClient()); + strings.deidentifyStringV2.mockRejectedValue(apiError(400, 'bad config', 'req-9')); + let thrown; + try { + await controller.deidentifyString(new DeidentifyStringRequest('Hi')); + } catch (e) { + thrown = e; + } + expect(thrown).toBeInstanceOf(SkyflowError); + expect(thrown.message).toBe('bad config'); + expect(thrown.error.httpCode).toBe(400); + expect(thrown.error.requestId).toBe('req-9'); + expect(thrown.error.grpcCode).toBe(3); + }); + + test('reidentifyString adds the vault id and maps the response', async () => { + const controller = new TestController(fakeClient()); + strings.reidentifyStringV2.mockResolvedValue({ processedText: 'Hi John', metrics: { characterCount: 7 } }); + const res = await controller.reidentifyString(new ReidentifyStringRequest('Hi [NAME_1]', { redactionLevel: [{ tokenGroupName: 'g' }] })); + expect(strings.reidentifyStringV2.mock.calls[0][0]).toEqual({ vaultId: 'vault123', text: 'Hi [NAME_1]', redactionLevel: [{ tokenGroupName: 'g' }] }); + expect(res).toBeInstanceOf(ReidentifyStringResponse); + expect(res.processedText).toBe('Hi John'); + }); +}); + +describe('deidentifyFile', () => { + let tmpDir; + let filePath; + beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'detect-controller-')); + filePath = path.join(tmpDir, 'notes.txt'); + fs.writeFileSync(filePath, 'John lives here'); + }); + afterEach(() => fs.rmSync(tmpDir, { recursive: true, force: true })); + + test('submit only: returns the run id, no polling, nothing written', async () => { + const controller = new TestController(fakeClient()); + files.deidentifyFileV2.mockResolvedValue({ runId: 'run-1' }); + const res = await controller.deidentifyFile(new DeidentifyFileRequest({ filePath, configurationId: 'cfg' })); + expect(res).toBeInstanceOf(DeidentifyFileResponse); + expect(res.runId).toBe('run-1'); + expect(res.status).toBeNull(); + expect(files.getRunV2).not.toHaveBeenCalled(); + const [sent] = files.deidentifyFileV2.mock.calls[0]; + expect(sent).toEqual({ dataSource: 'BASE64', value: Buffer.from('John lives here').toString('base64'), dataFormat: 'txt', configurationId: 'cfg' }); + expect(fs.readdirSync(tmpDir)).toEqual(['notes.txt']); + }); + + test('submit and poll: polls getRun until SUCCESS and writes outputs for a local file', async () => { + const controller = new TestController(fakeClient()); + files.deidentifyFileV2.mockResolvedValue({ runId: 'run-2' }); + files.getRunV2 + .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) + .mockResolvedValueOnce({ + status: 'SUCCESS', + outputType: 'BASE64', + output: [ + { processedFile: Buffer.from('[NAME_1] lives here').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: 'txt' }, + { processedFile: Buffer.from('[]').toString('base64'), processedFileType: 'ENTITIES', processedFileExtension: 'json' }, + ], + metrics: { wordCount: 3 }, + }); + const outDir = path.join(tmpDir, 'out'); + const res = await controller.deidentifyFile( + new DeidentifyFileRequest({ filePath, pollOptions: { waitTime: 30 }, outputDirectory: outDir }), + ); + expect(res.runId).toBe('run-2'); + expect(res.status).toBe(DetectRunStatus.SUCCESS); + expect(res.outputType).toBe(DeidentifySourceType.BASE64); + expect(res.output).toHaveLength(2); + expect(res.metrics.wordCount).toBe(3); + expect(files.getRunV2).toHaveBeenCalledTimes(2); + expect(files.getRunV2.mock.calls[0][0]).toBe('run-2'); + expect(files.getRunV2.mock.calls[0][1]).toEqual({ vaultId: 'vault123' }); + expect(fs.readFileSync(path.join(outDir, 'processed-notes.txt'), 'utf8')).toBe('[NAME_1] lives here'); + expect(fs.readFileSync(path.join(outDir, 'processed-entities-notes.json'), 'utf8')).toBe('[]'); + }); + + test('submit and poll on a value-based request writes nothing even on SUCCESS', async () => { + const controller = new TestController(fakeClient()); + files.deidentifyFileV2.mockResolvedValue({ runId: 'run-3' }); + files.getRunV2.mockResolvedValue({ + status: 'SUCCESS', + outputType: 'BASE64', + output: [{ processedFile: Buffer.from('x').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: 'txt' }], + }); + const res = await controller.deidentifyFile( + new DeidentifyFileRequest({ dataSource: DeidentifySourceType.SKYFLOW_ID, value: 'sid', pollOptions: {}, outputDirectory: tmpDir }), + ); + expect(res.status).toBe(DetectRunStatus.SUCCESS); + expect(fs.readdirSync(tmpDir)).toEqual(['notes.txt']); + }); + + test('a FAILED run returns the message without writing', async () => { + const controller = new TestController(fakeClient()); + files.deidentifyFileV2.mockResolvedValue({ runId: 'run-4' }); + files.getRunV2.mockResolvedValue({ status: 'FAILED', message: 'unsupported file' }); + const res = await controller.deidentifyFile(new DeidentifyFileRequest({ filePath, pollOptions: {} })); + expect(res.status).toBe(DetectRunStatus.FAILED); + expect(res.message).toBe('unsupported file'); + expect(fs.readdirSync(tmpDir)).toEqual(['notes.txt']); + }); + + test('a submit error is normalized', async () => { + const controller = new TestController(fakeClient()); + files.deidentifyFileV2.mockRejectedValue(apiError(413, 'too large')); + await expect(controller.deidentifyFile(new DeidentifyFileRequest({ filePath }))).rejects.toMatchObject({ message: 'too large' }); + }); + + test('a getRun failure while polling is normalized and propagated', async () => { + const controller = new TestController(fakeClient()); + files.deidentifyFileV2.mockResolvedValue({ runId: 'run-5' }); + files.getRunV2.mockRejectedValue(apiError(404, 'no such run')); + await expect(controller.deidentifyFile(new DeidentifyFileRequest({ filePath, pollOptions: {} }))).rejects.toMatchObject({ message: 'no such run' }); + }); +}); + +describe('reidentifyFile / getRun / checkGuardrails', () => { + test('reidentifyFile maps request and response', async () => { + const controller = new TestController(fakeClient()); + files.reidentifyFileV2.mockResolvedValue({ status: 'SUCCESS', outputType: 'BASE64', output: [{ processedFile: 'b64', processedFileType: 'REIDENTIFIED_FILE', processedFileExtension: 'txt' }] }); + const res = await controller.reidentifyFile(new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'abc')); + expect(files.reidentifyFileV2.mock.calls[0][0]).toEqual({ dataSource: 'BASE64', value: 'abc', vaultId: 'vault123' }); + expect(res).toBeInstanceOf(ReidentifyFileResponse); + expect(res.status).toBe(DetectRunStatus.SUCCESS); + expect(res.output[0].processedFile).toBe('b64'); + }); + + test('getRun trims the id, passes the vault id as the query, and maps the run', async () => { + const controller = new TestController(fakeClient()); + files.getRunV2.mockResolvedValue({ status: 'QUEUED' }); + const res = await controller.getRun(new GetRunRequest(' run-7 ')); + expect(files.getRunV2).toHaveBeenCalledWith('run-7', { vaultId: 'vault123' }, expect.any(Object)); + expect(res).toBeInstanceOf(GetRunResponse); + expect(res.runId).toBe('run-7'); + expect(res.status).toBe(DetectRunStatus.QUEUED); + }); + + test('checkGuardrails maps request and response', async () => { + const controller = new TestController(fakeClient()); + guardrails.checkGuardrailsV2.mockResolvedValue({ text: 't', validation: 'FAILED', toxic: true }); + const res = await controller.checkGuardrails(new CheckGuardrailsRequest('t', { checkToxicity: true, denyTopics: ['politics'] })); + expect(guardrails.checkGuardrailsV2.mock.calls[0][0]).toEqual({ text: 't', vaultId: 'vault123', checkToxicity: true, denyTopics: ['politics'] }); + expect(res).toBeInstanceOf(CheckGuardrailsResponse); + expect(res.validation).toBe(GuardrailsValidation.FAILED); + expect(res.toxic).toBe(true); + expect(res.deniedTopic).toBeNull(); + }); + + test('a generic Error from the transport becomes a SkyflowError with its message', async () => { + const controller = new TestController(fakeClient()); + guardrails.checkGuardrailsV2.mockRejectedValue(new Error('socket hang up')); + let thrown; + try { + await controller.checkGuardrails(new CheckGuardrailsRequest('t')); + } catch (e) { + thrown = e; + } + expect(thrown).toBeInstanceOf(SkyflowError); + expect(thrown.message).toBe('socket hang up'); + expect(thrown.error.httpCode).toBe(400); + }); + + test('the files client is built once and shared by file operations', async () => { + const controller = new TestController(fakeClient()); + files.getRunV2.mockResolvedValue({ status: 'SUCCESS' }); + await controller.getRun(new GetRunRequest('a')); + await controller.getRun(new GetRunRequest('b')); + expect(FilesV2).toHaveBeenCalledTimes(1); + }); +}); diff --git a/common/test/detect/edge-cases.test.js b/common/test/detect/edge-cases.test.js new file mode 100644 index 00000000..468d6f75 --- /dev/null +++ b/common/test/detect/edge-cases.test.js @@ -0,0 +1,215 @@ +// Covers the remaining branches of the detect layer: controller error paths for the +// re-identify operations, SkyflowError pass-through, the real-timer sleeper, output write +// failures, sparse media configurations, and default parameters. +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; + +jest.mock('../../src/_generated_/rest/api/resources/stringsV2/client/Client', () => { + const api = { deidentifyStringV2: jest.fn(), reidentifyStringV2: jest.fn() }; + return { StringsV2: jest.fn(() => api), __api: api }; +}); +jest.mock('../../src/_generated_/rest/api/resources/filesV2/client/Client', () => { + const api = { deidentifyFileV2: jest.fn(), reidentifyFileV2: jest.fn(), getRunV2: jest.fn() }; + return { FilesV2: jest.fn(() => api), __api: api }; +}); +jest.mock('../../src/_generated_/rest/api/resources/guardrailsV2/client/Client', () => { + const api = { checkGuardrailsV2: jest.fn() }; + return { GuardrailsV2: jest.fn(() => api), __api: api }; +}); + +import { __api as strings } from '../../src/_generated_/rest/api/resources/stringsV2/client/Client'; +import { __api as files } from '../../src/_generated_/rest/api/resources/filesV2/client/Client'; +import ReidentifyStringRequest from '../../src/detect/model/request/reidentify-string'; +import ReidentifyFileRequest from '../../src/detect/model/request/reidentify-file'; +import DeidentifyFileRequest from '../../src/detect/model/request/deidentify-file'; +import GetRunResponse from '../../src/detect/model/response/get-run'; +import { toSkyflowError } from '../../src/detect/internal/errors'; +import { pollRun } from '../../src/detect/internal/run-poller'; +import { writeOutputs } from '../../src/detect/internal/file-output-writer'; +import { toDetectConfig } from '../../src/detect/internal/request-mapper'; +import { completeDeidentifyFile } from '../../src/detect/internal/deidentify-file-flow'; +import { DetectRunStatus, FileDataFormat, ObjectEntityType } from '../../src/detect/types'; +import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType, TestDetectController } from './test-types'; +import { LogLevel } from '../../src/utils'; +import SkyflowError from '../../src/error'; +import SKYFLOW_ERROR_CODE from '../../src/error/codes'; + +beforeAll(() => { + jest.spyOn(console, 'error').mockImplementation(() => undefined); + jest.spyOn(console, 'log').mockImplementation(() => undefined); +}); +afterAll(() => jest.restoreAllMocks()); +beforeEach(() => jest.clearAllMocks()); + +const fakeClient = () => ({ + vaultId: 'vault123', + url: 'https://abc.vault.skyflowapis.com', + getLogLevel: () => LogLevel.ERROR, + resolveAuthToken: jest.fn().mockResolvedValue('tok'), +}); + +describe('controller error paths', () => { + test('reidentifyString normalizes an API failure', async () => { + const controller = new TestDetectController(fakeClient()); + strings.reidentifyStringV2.mockRejectedValue({ statusCode: 403, body: { error: { message: 'forbidden' } } }); + await expect(controller.reidentifyString(new ReidentifyStringRequest('[NAME_1]'))).rejects.toMatchObject({ + message: 'forbidden', + error: expect.objectContaining({ httpCode: 403 }), + }); + }); + + test('reidentifyFile normalizes an API failure', async () => { + const controller = new TestDetectController(fakeClient()); + files.reidentifyFileV2.mockRejectedValue({ statusCode: 500, body: { error: { message: 'boom' } } }); + await expect(controller.reidentifyFile(new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'abc'))).rejects.toMatchObject({ + message: 'boom', + }); + }); + + test('a SkyflowError thrown by the transport passes through unchanged', async () => { + const controller = new TestDetectController(fakeClient()); + const original = new SkyflowError(SKYFLOW_ERROR_CODE.INTERNAL_SERVER_ERROR, ['x']); + strings.reidentifyStringV2.mockRejectedValue(original); + await expect(controller.reidentifyString(new ReidentifyStringRequest('t'))).rejects.toBe(original); + expect(toSkyflowError(original, 'log')).toBe(original); + }); + + test('the base controller uses the real timer sleeper by default', () => { + expect(new TestDetectController(fakeClient()).sleeper()).toBeUndefined(); + }); +}); + +describe('run poller with the real timer', () => { + test('waits on setTimeout between lookups when no sleeper is supplied', async () => { + jest.useFakeTimers(); + try { + const fetcher = jest + .fn() + .mockResolvedValueOnce(new GetRunResponse({ runId: 'r', status: DetectRunStatus.IN_PROGRESS, outputType: null, output: [], metrics: null, message: null })) + .mockResolvedValueOnce(new GetRunResponse({ runId: 'r', status: DetectRunStatus.SUCCESS, outputType: null, output: [], metrics: null, message: null })); + const pending = pollRun('r', { waitTime: 5, maxAttempts: 5 }, fetcher); + await jest.advanceTimersByTimeAsync(1000); + const out = await pending; + expect(out.status).toBe(DetectRunStatus.SUCCESS); + expect(fetcher).toHaveBeenCalledTimes(2); + } finally { + jest.useRealTimers(); + } + }); +}); + +describe('deidentify-file flow defaults', () => { + test('completeDeidentifyFile submit-only works with default log level and sleeper', async () => { + const res = await completeDeidentifyFile(new DeidentifyFileRequest({ filePath: '/x' }), 'run-1', jest.fn()); + expect(res.runId).toBe('run-1'); + expect(res.status).toBeNull(); + }); +}); + +describe('output writer failures', () => { + test('throws FAILED_TO_WRITE_OUTPUT_FILE when a target cannot be written', async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'detect-writer-fail-')); + try { + // A directory already occupies the output file name, so writeFile fails. + fs.mkdirSync(path.join(tmpDir, 'processed-notes.txt')); + let thrown; + try { + await writeOutputs( + [{ processedFile: Buffer.from('a').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }], + DeidentifySourceType.BASE64, + 'notes.txt', + tmpDir, + ); + } catch (e) { + thrown = e; + } + expect(thrown).toBeInstanceOf(SkyflowError); + expect(thrown.message).toContain('processed-notes.txt'); + } finally { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } + }); +}); + +describe('submit without a run id', () => { + test('deidentifyFile returns a null run id when the API omits it, and does not poll', async () => { + const controller = new TestDetectController(fakeClient()); + files.deidentifyFileV2.mockResolvedValue({}); + const res = await controller.deidentifyFile( + new DeidentifyFileRequest({ dataSource: DeidentifySourceType.PRESIGNED_URL, value: 'https://u', pollOptions: {} }), + ); + expect(res.runId).toBeNull(); + expect(res.status).toBeNull(); + expect(files.getRunV2).not.toHaveBeenCalled(); + }); +}); + +describe('toSkyflowError fallbacks', () => { + test('a null error yields the generic message and a 400 code', () => { + const err = toSkyflowError(null, 'log'); + expect(err).toBeInstanceOf(SkyflowError); + expect(err.error.httpCode).toBe(400); + expect(err.error.requestId).toBeNull(); + }); + + test('camelCase body fields and a body request id are honoured; a rawResponse without headers is tolerated', () => { + const err = toSkyflowError( + { + body: { error: { message: 'm', httpCode: 429, grpcCode: 8, httpStatus: 'Too Many Requests', details: ['d'], requestId: 'body-req' } }, + rawResponse: {}, + }, + 'log', + ); + expect(err.error.httpCode).toBe(429); + expect(err.error.grpcCode).toBe(8); + expect(err.error.httpStatus).toBe('Too Many Requests'); + expect(err.error.details).toEqual(['d']); + expect(err.error.requestId).toBe('body-req'); + }); + + test('a plain Error with no body uses its message; a non-object body is ignored', () => { + expect(toSkyflowError(new Error('socket'), 'log').message).toBe('socket'); + const err = toSkyflowError({ statusCode: 502, body: 'gateway text', rawResponse: { headers: new Map() } }, 'log'); + expect(err.error.httpCode).toBe(502); + expect(err.error.requestId).toBeNull(); + }); +}); + +describe('output writer directory defaults', () => { + test('a blank outputDirectory writes into the working directory', async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'detect-writer-cwd-')); + const previous = process.cwd(); + process.chdir(tmpDir); + try { + const written = await writeOutputs( + [{ processedFile: Buffer.from('a').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }], + DeidentifySourceType.BASE64, + 'notes.txt', + ' ', + ); + expect(written).toEqual(['processed-notes.txt']); + expect(fs.existsSync(path.join(tmpDir, 'processed-notes.txt'))).toBe(true); + } finally { + process.chdir(previous); + fs.rmSync(tmpDir, { recursive: true, force: true }); + } + }); +}); + +describe('sparse configuration mapping', () => { + test('audio without bleep, document without pdf, and null object entities map cleanly', () => { + const out = toDetectConfig( + { + detect: { objectEntities: [null, { entityType: ObjectEntityType.LOGO }] }, + media: { audio: { outputProcessedAudio: false }, document: {} }, + }, + 'vault123', + ); + expect(out).toEqual({ + vaultId: 'vault123', + detect: { objectEntities: [{ entityType: 'LOGO' }] }, + media: { audio: { outputProcessedAudio: false }, document: {} }, + }); + }); +}); diff --git a/common/test/detect/file-output-writer.test.js b/common/test/detect/file-output-writer.test.js new file mode 100644 index 00000000..4878212c --- /dev/null +++ b/common/test/detect/file-output-writer.test.js @@ -0,0 +1,80 @@ +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { writeOutputs, outputFileName, stripExtension } from '../../src/detect/internal/file-output-writer'; +import { FileDataFormat } from '../../src/detect/types'; +import { TestDeidentifySourceType as DeidentifySourceType } from './test-types'; +import SkyflowError from '../../src/error'; + +beforeAll(() => { + jest.spyOn(console, 'error').mockImplementation(() => undefined); + jest.spyOn(console, 'log').mockImplementation(() => undefined); +}); +afterAll(() => jest.restoreAllMocks()); + +const b64 = (s) => Buffer.from(s).toString('base64'); + +describe('stripExtension', () => { + test('drops the last extension, keeps dot-less names, defaults empty to "output"', () => { + expect(stripExtension('notes.pdf')).toBe('notes'); + expect(stripExtension('archive.tar.gz')).toBe('archive.tar'); + expect(stripExtension('README')).toBe('README'); + expect(stripExtension('')).toBe('output'); + expect(stripExtension(undefined)).toBe('output'); + }); +}); + +describe('outputFileName', () => { + test('names processed files and entity files, and avoids collisions', () => { + const used = new Set(); + expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.PDF }, used)).toBe('processed-notes.pdf'); + expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'ENTITIES', processedFileExtension: FileDataFormat.JSON }, used)).toBe('processed-entities-notes.json'); + expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'OBJECT_ENTITIES', processedFileExtension: null }, used)).toBe('processed-object-entities-notes.json'); + // Same stem + extension again -> type inserted before the extension. + expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'REDACTED_TEXT', processedFileExtension: FileDataFormat.PDF }, used)).toBe('processed-notes.redacted_text.pdf'); + // No type and no extension -> .bin, and a counter on collision. + expect(outputFileName('notes', { processedFile: 'x', processedFileType: null, processedFileExtension: null }, used)).toBe('processed-notes.bin'); + expect(outputFileName('notes', { processedFile: 'x', processedFileType: null, processedFileExtension: null }, used)).toBe('processed-notes.5.bin'); + }); +}); + +describe('writeOutputs', () => { + let tmpDir; + beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'detect-writer-')); + }); + afterEach(() => fs.rmSync(tmpDir, { recursive: true, force: true })); + + test('writes decoded BASE64 outputs into the directory, creating it when missing', async () => { + const dir = path.join(tmpDir, 'nested', 'out'); + const written = await writeOutputs( + [ + { processedFile: b64('redacted'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }, + { processedFile: b64('[]'), processedFileType: 'ENTITIES', processedFileExtension: FileDataFormat.JSON }, + { processedFile: null, processedFileType: 'REDACTED_TEXT', processedFileExtension: FileDataFormat.TXT }, + ], + DeidentifySourceType.BASE64, + 'notes.txt', + dir, + ); + expect(written).toEqual([path.join(dir, 'processed-notes.txt'), path.join(dir, 'processed-entities-notes.json')]); + expect(fs.readFileSync(written[0], 'utf8')).toBe('redacted'); + expect(fs.readFileSync(written[1], 'utf8')).toBe('[]'); + }); + + test('returns nothing for empty outputs or non-BASE64 output types', async () => { + expect(await writeOutputs([], DeidentifySourceType.BASE64, 'a.txt', tmpDir)).toEqual([]); + expect( + await writeOutputs([{ processedFile: 'https://u', processedFileType: 'REDACTED_FILE', processedFileExtension: null }], DeidentifySourceType.PRESIGNED_URL, 'a.txt', tmpDir), + ).toEqual([]); + expect(fs.readdirSync(tmpDir)).toEqual([]); + }); + + test('throws a SkyflowError when the directory cannot be created', async () => { + const blocker = path.join(tmpDir, 'file-not-dir'); + fs.writeFileSync(blocker, 'x'); + await expect( + writeOutputs([{ processedFile: b64('a'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }], DeidentifySourceType.BASE64, 'a.txt', path.join(blocker, 'sub')), + ).rejects.toBeInstanceOf(SkyflowError); + }); +}); diff --git a/common/test/detect/models.test.js b/common/test/detect/models.test.js new file mode 100644 index 00000000..f73355ef --- /dev/null +++ b/common/test/detect/models.test.js @@ -0,0 +1,123 @@ +import { + DeidentifyStringRequest, + DeidentifyFileRequest, + ReidentifyStringRequest, + ReidentifyFileRequest, + GetRunRequest, + CheckGuardrailsRequest, + DeidentifyStringResponse, + DeidentifyFileResponse, + ReidentifyStringResponse, + ReidentifyFileResponse, + GetRunResponse, + CheckGuardrailsResponse, + DetectRunStatus, + EntityType, + FileDataFormat, + GuardrailsValidation, + POLL_OPTIONS, + BaseDetectController, +} from '../../src'; +import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType } from './test-types'; + +describe('detect request models', () => { + test('DeidentifyStringRequest stores text and optional fields with settable accessors', () => { + const r = new DeidentifyStringRequest('t', { configurationId: 'c' }); + expect(r.text).toBe('t'); + expect(r.configurationId).toBe('c'); + expect(r.configuration).toBeUndefined(); + r.text = 'u'; + r.configurationId = undefined; + r.configuration = { detect: { entities: [{ entityType: EntityType.NAME }] } }; + expect(r.text).toBe('u'); + expect(r.configurationId).toBeUndefined(); + expect(r.configuration.detect.entities[0].entityType).toBe('NAME'); + }); + + test('DeidentifyFileRequest stores every field and defaults to an empty request', () => { + const empty = new DeidentifyFileRequest(); + expect(empty.dataSource).toBeUndefined(); + expect(empty.pollOptions).toBeUndefined(); + + const r = new DeidentifyFileRequest({ + dataSource: DeidentifySourceType.PRESIGNED_URL, + value: 'https://u', + dataFormat: FileDataFormat.PDF, + configurationId: 'c', + pollOptions: { waitTime: 10 }, + outputDirectory: '/out', + }); + expect(r.dataSource).toBe(DeidentifySourceType.PRESIGNED_URL); + expect(r.value).toBe('https://u'); + expect(r.dataFormat).toBe(FileDataFormat.PDF); + expect(r.configurationId).toBe('c'); + expect(r.pollOptions).toEqual({ waitTime: 10 }); + expect(r.outputDirectory).toBe('/out'); + + r.dataSource = DeidentifySourceType.BASE64; + r.value = 'abc'; + r.file = undefined; + r.filePath = '/in/a.txt'; + r.dataFormat = undefined; + r.configurationId = undefined; + r.configuration = { name: 'n' }; + r.pollOptions = undefined; + r.outputDirectory = undefined; + expect(r.filePath).toBe('/in/a.txt'); + expect(r.configuration.name).toBe('n'); + expect(r.file).toBeUndefined(); + }); + + test('ReidentifyStringRequest / ReidentifyFileRequest / GetRunRequest / CheckGuardrailsRequest', () => { + const s = new ReidentifyStringRequest('t', { redactionLevel: [{ tokenGroupName: 'g' }] }); + expect(s.redactionLevel).toEqual([{ tokenGroupName: 'g' }]); + s.text = 'x'; + s.redactionLevel = undefined; + expect(s.text).toBe('x'); + expect(s.redactionLevel).toBeUndefined(); + + const f = new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'v', { dataFormat: FileDataFormat.TXT }); + expect(f.dataSource).toBe(ReidentifySourceType.BASE64); + expect(f.value).toBe('v'); + expect(f.dataFormat).toBe(FileDataFormat.TXT); + expect(f.redactionLevel).toBeUndefined(); + f.dataSource = ReidentifySourceType.PRESIGNED_URL; + f.value = 'w'; + f.dataFormat = undefined; + f.redactionLevel = []; + expect(f.dataSource).toBe(ReidentifySourceType.PRESIGNED_URL); + expect(f.redactionLevel).toEqual([]); + + const g = new GetRunRequest('run'); + g.runId = 'run2'; + expect(g.runId).toBe('run2'); + + const c = new CheckGuardrailsRequest('t', { checkToxicity: true, denyTopics: ['a'] }); + expect(c.checkToxicity).toBe(true); + expect(c.denyTopics).toEqual(['a']); + c.text = 'u'; + c.checkToxicity = undefined; + c.denyTopics = undefined; + expect(c.text).toBe('u'); + expect(c.checkToxicity).toBeUndefined(); + }); +}); + +describe('detect response models', () => { + test('responses are plain readonly data containers', () => { + expect(new DeidentifyStringResponse({ processedText: 'p', entities: [], metrics: null })).toEqual( + expect.objectContaining({ processedText: 'p', entities: [], metrics: null }), + ); + expect(new ReidentifyStringResponse({ processedText: 'p', metrics: null }).processedText).toBe('p'); + const run = { runId: 'r', status: DetectRunStatus.SUCCESS, outputType: null, output: [], metrics: null, message: null }; + expect(new DeidentifyFileResponse(run)).toEqual(expect.objectContaining(run)); + expect(new GetRunResponse(run)).toEqual(expect.objectContaining(run)); + expect(new ReidentifyFileResponse({ status: DetectRunStatus.FAILED, outputType: null, output: [], metrics: null }).status).toBe(DetectRunStatus.FAILED); + expect(new CheckGuardrailsResponse({ text: 't', validation: GuardrailsValidation.PASSED, toxic: null, deniedTopic: null }).validation).toBe('PASSED'); + }); + + test('the public barrel exposes the poll limits and the base controller', () => { + expect(POLL_OPTIONS.MAX_WAIT_TIME_SECONDS).toBe(300); + expect(typeof BaseDetectController).toBe('function'); + }); +}); diff --git a/common/test/detect/request-mapper.test.js b/common/test/detect/request-mapper.test.js new file mode 100644 index 00000000..31bfa7db --- /dev/null +++ b/common/test/detect/request-mapper.test.js @@ -0,0 +1,254 @@ +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { + sendsConfiguration, + toDeidentifyStringRequest, + toDeidentifyFileRequest, + toReidentifyStringRequest, + toReidentifyFileRequest, + toCheckGuardrailsRequest, + toDetectConfig, + toRedactionLevels, + inferFormat, + localFileName, +} from '../../src/detect/internal/request-mapper'; +import DeidentifyStringRequest from '../../src/detect/model/request/deidentify-string'; +import DeidentifyFileRequest from '../../src/detect/model/request/deidentify-file'; +import ReidentifyStringRequest from '../../src/detect/model/request/reidentify-string'; +import ReidentifyFileRequest from '../../src/detect/model/request/reidentify-file'; +import CheckGuardrailsRequest from '../../src/detect/model/request/check-guardrails'; +import { + DeidentificationType, + EntityType, + FileDataFormat, + MaskingMethod, + ObjectDeidentificationType, + ObjectEntityType, + OutputTranscriptionType, + PdfProcessingMode, + ReturnEntitiesType, +} from '../../src/detect/types'; +import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType } from './test-types'; +import { RedactionType } from '../../src/utils'; +import SkyflowError from '../../src/error'; +import SKYFLOW_ERROR_CODE from '../../src/error/codes'; + +beforeAll(() => { + jest.spyOn(console, 'error').mockImplementation(() => undefined); +}); +afterAll(() => jest.restoreAllMocks()); + +const VAULT = 'vault123'; + +describe('sendsConfiguration', () => { + test('sends when inline configuration is set, or when neither is set', () => { + expect(sendsConfiguration(undefined, undefined)).toBe(true); + expect(sendsConfiguration(undefined, { detect: {} })).toBe(true); + expect(sendsConfiguration('cfg', { detect: {} })).toBe(true); + expect(sendsConfiguration('cfg', undefined)).toBe(false); + }); +}); + +describe('toDeidentifyStringRequest', () => { + test('with neither id nor configuration, sends a configuration carrying only the vault id', () => { + expect(toDeidentifyStringRequest(new DeidentifyStringRequest('hello'), VAULT)).toEqual({ + text: 'hello', + configuration: { vaultId: VAULT }, + }); + }); + test('with only a configuration id, sends the id and nothing else', () => { + expect(toDeidentifyStringRequest(new DeidentifyStringRequest('hello', { configurationId: 'cfg' }), VAULT)).toEqual({ + text: 'hello', + configurationId: 'cfg', + }); + }); + test('with both, sends both (the server rejects the combination)', () => { + const out = toDeidentifyStringRequest( + new DeidentifyStringRequest('hello', { configurationId: 'cfg', configuration: { name: 'n' } }), + VAULT, + ); + expect(out.configurationId).toBe('cfg'); + expect(out.configuration).toEqual({ vaultId: VAULT, name: 'n' }); + }); +}); + +describe('toDetectConfig', () => { + test('maps every field of a full inline configuration and drops unset keys', () => { + const out = toDetectConfig( + { + name: 'cfg', + description: 'desc', + detect: { + entities: [ + { + entityType: EntityType.DOB, + deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, + transformation: { shiftDates: { minDays: 10, maxDays: 30 } }, + }, + { entityType: EntityType.NAME, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: 'pii.name' }, + null, + ], + objectEntities: [{ entityType: ObjectEntityType.FACE, deidentificationType: ObjectDeidentificationType.REDACT }], + restrict: ['[0-9]{3}'], + skip: ['Skyflow'], + customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], + returnEntities: ReturnEntitiesType.ALL, + }, + fileMapping: [{ source: 'a', destination: 'b', outputFileNameSuffix: '_x', entitiesDestination: 'e', objectEntitiesDestination: 'o' }], + media: { + audio: { outputProcessedAudio: true, outputTranscription: OutputTranscriptionType.TRANSCRIPTION, bleep: { startPadding: 0.5, gain: 2 } }, + document: { pdf: { processingMode: PdfProcessingMode.OCR, density: 300 } }, + image: { maskingMethod: MaskingMethod.BLUR, outputOcrText: true }, + }, + }, + VAULT, + ); + expect(out).toEqual({ + vaultId: VAULT, + name: 'cfg', + description: 'desc', + detect: { + entities: [ + { entityType: 'DOB', deidentificationType: 'ENTITY_UNIQUE_COUNTER', transformation: { shiftDates: { minDays: 10, maxDays: 30 } } }, + { entityType: 'NAME', deidentificationType: 'VAULT_TOKEN', destination: 'pii.name' }, + ], + objectEntities: [{ entityType: 'FACE', deidentificationType: 'REDACT' }], + restrict: ['[0-9]{3}'], + skip: ['Skyflow'], + customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], + returnEntities: 'ALL', + }, + fileMapping: [{ source: 'a', destination: 'b', outputFileNameSuffix: '_x', entitiesDestination: 'e', objectEntitiesDestination: 'o' }], + media: { + audio: { outputProcessedAudio: true, outputTranscription: 'TRANSCRIPTION', bleep: { startPadding: 0.5, gain: 2 } }, + document: { pdf: { processingMode: 'OCR', density: 300 } }, + image: { maskingMethod: 'BLUR', outputOcrText: true }, + }, + }); + }); + test('an empty transformation / media / detect maps to empty objects', () => { + const out = toDetectConfig({ detect: { entities: [{ transformation: {} }] }, media: {} }, VAULT); + expect(out).toEqual({ vaultId: VAULT, detect: { entities: [{ transformation: {} }] }, media: {} }); + }); +}); + +describe('toDeidentifyFileRequest', () => { + let tmpDir; + let filePath; + beforeAll(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'detect-mapper-')); + filePath = path.join(tmpDir, 'patient-notes.pdf'); + fs.writeFileSync(filePath, 'pdf-bytes'); + }); + afterAll(() => fs.rmSync(tmpDir, { recursive: true, force: true })); + + test('value-based request passes dataSource, value and lower-cased dataFormat through', async () => { + const out = await toDeidentifyFileRequest( + new DeidentifyFileRequest({ + dataSource: DeidentifySourceType.SKYFLOW_ID, + value: 'sid', + dataFormat: FileDataFormat.DOCX, + configurationId: 'cfg', + }), + VAULT, + ); + expect(out).toEqual({ dataSource: 'SKYFLOW_ID', value: 'sid', dataFormat: 'docx', configurationId: 'cfg' }); + }); + test('filePath request is read, base64-encoded, sent as BASE64 and the format inferred', async () => { + const out = await toDeidentifyFileRequest(new DeidentifyFileRequest({ filePath }), VAULT); + expect(out.dataSource).toBe('BASE64'); + expect(out.value).toBe(Buffer.from('pdf-bytes').toString('base64')); + expect(out.dataFormat).toBe('pdf'); + expect(out.configuration).toEqual({ vaultId: VAULT }); + }); + test('File request is encoded from arrayBuffer and an explicit dataFormat wins over inference', async () => { + const file = { name: 'x.txt', arrayBuffer: async () => Uint8Array.from(Buffer.from('hello')).buffer }; + const out = await toDeidentifyFileRequest( + new DeidentifyFileRequest({ file, dataFormat: FileDataFormat.CSV, dataSource: DeidentifySourceType.PRESIGNED_URL }), + VAULT, + ); + expect(out.dataSource).toBe('BASE64'); // local file always wins + expect(out.value).toBe(Buffer.from('hello').toString('base64')); + expect(out.dataFormat).toBe('csv'); + }); + test('an unknown extension leaves dataFormat unset so the server infers it', async () => { + const unknown = path.join(tmpDir, 'blob.weird'); + fs.writeFileSync(unknown, 'x'); + const out = await toDeidentifyFileRequest(new DeidentifyFileRequest({ filePath: unknown }), VAULT); + expect(out.dataFormat).toBeUndefined(); + }); + test('an unreadable file throws FAILED_TO_ENCODE_FILE', async () => { + const file = { name: 'bad.txt', arrayBuffer: async () => { throw new Error('boom'); } }; + let thrown; + try { + await toDeidentifyFileRequest(new DeidentifyFileRequest({ file }), VAULT); + } catch (e) { + thrown = e; + } + expect(thrown).toBeInstanceOf(SkyflowError); + expect(thrown.message).toBe(SKYFLOW_ERROR_CODE.FAILED_TO_ENCODE_FILE.message); + }); +}); + +describe('inferFormat / localFileName', () => { + test('inferFormat maps known extensions case-insensitively and returns undefined otherwise', () => { + expect(inferFormat('a.PDF')).toBe(FileDataFormat.PDF); + expect(inferFormat('a.jsonl')).toBe(FileDataFormat.JSONL); + expect(inferFormat('a.unknown')).toBeUndefined(); + expect(inferFormat('noext')).toBeUndefined(); + expect(inferFormat(null)).toBeUndefined(); + }); + test('localFileName uses the File name, the filePath basename, or null', () => { + expect(localFileName(new DeidentifyFileRequest({ file: { name: '/tmp/x/a.txt', arrayBuffer: async () => null } }))).toBe('a.txt'); + expect(localFileName(new DeidentifyFileRequest({ file: { arrayBuffer: async () => null } }))).toBe('output'); + expect(localFileName(new DeidentifyFileRequest({ filePath: '/data/in/notes.pdf' }))).toBe('notes.pdf'); + expect(localFileName(new DeidentifyFileRequest({ dataSource: DeidentifySourceType.BASE64, value: 'x' }))).toBeNull(); + }); +}); + +describe('reidentify / guardrails mappers', () => { + test('toReidentifyStringRequest adds the vault id and maps redaction levels', () => { + const out = toReidentifyStringRequest( + new ReidentifyStringRequest('[NAME_1]', { + redactionLevel: [ + { tokenGroupName: 'g', redactionPattern: 'p' }, + { entityName: EntityType.SSN, redactionType: RedactionType.MASKED }, + null, + ], + }), + VAULT, + ); + expect(out).toEqual({ + vaultId: VAULT, + text: '[NAME_1]', + redactionLevel: [ + { tokenGroupName: 'g', redactionPattern: 'p' }, + { entityName: 'SSN', redactionType: 'MASKED' }, + ], + }); + expect(toReidentifyStringRequest(new ReidentifyStringRequest('t'), VAULT)).toEqual({ vaultId: VAULT, text: 't' }); + }); + test('toReidentifyFileRequest lower-cases the format and maps redaction levels', () => { + const out = toReidentifyFileRequest( + new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'abc', { + dataFormat: FileDataFormat.TXT, + redactionLevel: [{ tokenGroupName: 'g' }], + }), + VAULT, + ); + expect(out).toEqual({ dataSource: 'BASE64', value: 'abc', vaultId: VAULT, dataFormat: 'txt', redactionLevel: [{ tokenGroupName: 'g' }] }); + }); + test('toCheckGuardrailsRequest sends optional checks only when set', () => { + expect(toCheckGuardrailsRequest(new CheckGuardrailsRequest('t'), VAULT)).toEqual({ text: 't', vaultId: VAULT }); + expect(toCheckGuardrailsRequest(new CheckGuardrailsRequest('t', { checkToxicity: false, denyTopics: ['a'] }), VAULT)).toEqual({ + text: 't', + vaultId: VAULT, + checkToxicity: false, + denyTopics: ['a'], + }); + }); + test('toRedactionLevels drops undefined keys', () => { + expect(toRedactionLevels([{ tokenGroupName: 'g', entityName: undefined }])).toEqual([{ tokenGroupName: 'g' }]); + }); +}); diff --git a/common/test/detect/response-mapper.test.js b/common/test/detect/response-mapper.test.js new file mode 100644 index 00000000..56d7c919 --- /dev/null +++ b/common/test/detect/response-mapper.test.js @@ -0,0 +1,150 @@ +import { + enumOrDefault, + toMetrics, + toDeidentifyStringResponse, + toReidentifyStringResponse, + toFileOutput, + toGetRunResponse as mapGetRunResponse, + submittedDeidentifyFileResponse, + toDeidentifyFileResponse, + toReidentifyFileResponse as mapReidentifyFileResponse, + toCheckGuardrailsResponse, +} from '../../src/detect/internal/response-mapper'; +import { DetectRunStatus, FileDataFormat, GuardrailsValidation } from '../../src/detect/types'; +import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType, resolveDeidentifySource, resolveReidentifySource } from './test-types'; + +// The mappers take the SDK's source resolver; these bind the test stand-ins. +const toGetRunResponse = (runId, response) => mapGetRunResponse(runId, response, resolveDeidentifySource); +const toReidentifyFileResponse = (response) => mapReidentifyFileResponse(response, resolveReidentifySource); +import DeidentifyStringResponse from '../../src/detect/model/response/deidentify-string'; +import GetRunResponse from '../../src/detect/model/response/get-run'; +import DeidentifyFileResponse from '../../src/detect/model/response/deidentify-file'; + +describe('enumOrDefault', () => { + test('returns the matching enum member, else the fallback', () => { + expect(enumOrDefault(DetectRunStatus, 'SUCCESS', DetectRunStatus.UNKNOWN)).toBe(DetectRunStatus.SUCCESS); + expect(enumOrDefault(DetectRunStatus, 'NEW_STATUS', DetectRunStatus.UNKNOWN)).toBe(DetectRunStatus.UNKNOWN); + expect(enumOrDefault(DetectRunStatus, undefined, null)).toBeNull(); + }); +}); + +describe('toMetrics', () => { + test('maps every field, null-filling missing ones', () => { + expect(toMetrics({ size: 1.5, wordCount: 3 })).toEqual({ + size: 1.5, + wordCount: 3, + characterCount: null, + slides: null, + pages: null, + duration: null, + }); + expect(toMetrics(undefined)).toBeNull(); + }); +}); + +describe('string responses', () => { + test('toDeidentifyStringResponse maps entities, locations and scores; entities never null', () => { + const out = toDeidentifyStringResponse({ + processedText: 'Hi [NAME_1]', + entities: [ + { + token: 'NAME_1', + value: 'John', + location: { startIndex: 3, endIndex: 7 }, + entityType: 'NAME', + entityScores: { NAME: 0.98 }, + }, + null, + { token: 'X' }, + ], + metrics: { wordCount: 2 }, + }); + expect(out).toBeInstanceOf(DeidentifyStringResponse); + expect(out.processedText).toBe('Hi [NAME_1]'); + expect(out.entities).toEqual([ + { + token: 'NAME_1', + value: 'John', + location: { startIndex: 3, endIndex: 7, startIndexProcessed: null, endIndexProcessed: null }, + entityType: 'NAME', + entityScores: { NAME: 0.98 }, + }, + { token: 'X', value: null, location: null, entityType: null, entityScores: {} }, + ]); + expect(out.metrics.wordCount).toBe(2); + expect(toDeidentifyStringResponse({}).entities).toEqual([]); + expect(toDeidentifyStringResponse({}).processedText).toBeNull(); + }); + test('toReidentifyStringResponse maps text and metrics', () => { + const out = toReidentifyStringResponse({ processedText: 'Hi John', metrics: { characterCount: 7 } }); + expect(out.processedText).toBe('Hi John'); + expect(out.metrics.characterCount).toBe(7); + }); +}); + +describe('file responses', () => { + test('toFileOutput upper-cases the extension into FileDataFormat and tolerates unknowns', () => { + expect(toFileOutput({ processedFile: 'b64', processedFileType: 'REDACTED_FILE', processedFileExtension: 'pdf' })).toEqual({ + processedFile: 'b64', + processedFileType: 'REDACTED_FILE', + processedFileExtension: FileDataFormat.PDF, + }); + expect(toFileOutput({ processedFileExtension: 'weird' })).toEqual({ + processedFile: null, + processedFileType: null, + processedFileExtension: null, + }); + }); + test('toGetRunResponse carries the run id and maps status/outputType with UNKNOWN fallback', () => { + const out = toGetRunResponse('run-1', { + status: 'SUCCESS', + outputType: 'BASE64', + output: [{ processedFile: 'x', processedFileType: 'ENTITIES', processedFileExtension: 'json' }, null], + metrics: { pages: 2 }, + message: undefined, + }); + expect(out).toBeInstanceOf(GetRunResponse); + expect(out.runId).toBe('run-1'); + expect(out.status).toBe(DetectRunStatus.SUCCESS); + expect(out.outputType).toBe(DeidentifySourceType.BASE64); + expect(out.output).toHaveLength(1); + expect(out.metrics.pages).toBe(2); + expect(out.message).toBeNull(); + + const unknown = toGetRunResponse('run-2', { status: 'SOMETHING_NEW' }); + expect(unknown.status).toBe(DetectRunStatus.UNKNOWN); + expect(unknown.outputType).toBeNull(); + expect(unknown.output).toEqual([]); + expect(unknown.metrics).toBeNull(); + }); + test('submittedDeidentifyFileResponse has only the run id; toDeidentifyFileResponse merges a run', () => { + const submitted = submittedDeidentifyFileResponse('run-1'); + expect(submitted).toBeInstanceOf(DeidentifyFileResponse); + expect(submitted).toEqual(expect.objectContaining({ runId: 'run-1', status: null, outputType: null, output: [], metrics: null, message: null })); + + const run = toGetRunResponse('run-1', { status: 'FAILED', message: 'bad file' }); + const merged = toDeidentifyFileResponse('run-1', run); + expect(merged.status).toBe(DetectRunStatus.FAILED); + expect(merged.message).toBe('bad file'); + expect(merged.runId).toBe('run-1'); + }); + test('toReidentifyFileResponse maps status, output type and outputs', () => { + const out = toReidentifyFileResponse({ status: 'SUCCESS', outputType: 'PRESIGNED_URL', output: [{ processedFile: 'https://u' }] }); + expect(out.status).toBe(DetectRunStatus.SUCCESS); + expect(out.outputType).toBe(ReidentifySourceType.PRESIGNED_URL); + expect(out.output[0].processedFile).toBe('https://u'); + expect(toReidentifyFileResponse({}).status).toBe(DetectRunStatus.UNKNOWN); + }); +}); + +describe('toCheckGuardrailsResponse', () => { + test('passes the API values through, with UNKNOWN for an unrecognized validation', () => { + expect(toCheckGuardrailsResponse({ text: 't', validation: 'FAILED', toxic: true, deniedTopic: false })).toEqual( + expect.objectContaining({ text: 't', validation: GuardrailsValidation.FAILED, toxic: true, deniedTopic: false }), + ); + const sparse = toCheckGuardrailsResponse({ text: 't', validation: 'MAYBE' }); + expect(sparse.validation).toBe(GuardrailsValidation.UNKNOWN); + expect(sparse.toxic).toBeNull(); + expect(sparse.deniedTopic).toBeNull(); + }); +}); diff --git a/common/test/detect/run-poller.test.js b/common/test/detect/run-poller.test.js new file mode 100644 index 00000000..2002b386 --- /dev/null +++ b/common/test/detect/run-poller.test.js @@ -0,0 +1,91 @@ +import { pollRun, effectiveWaitTime, effectiveMaxAttempts } from '../../src/detect/internal/run-poller'; +import GetRunResponse from '../../src/detect/model/response/get-run'; +import { DetectRunStatus, POLL_OPTIONS } from '../../src/detect/types'; + +const run = (status, extra = {}) => + new GetRunResponse({ runId: 'run-1', status, outputType: null, output: [], metrics: null, message: null, ...extra }); + +/** A fetcher that returns the given statuses in order, then repeats the last one. */ +function sequence(statuses) { + let i = 0; + const fetcher = jest.fn(async () => { + const s = statuses[Math.min(i, statuses.length - 1)]; + i += 1; + return s === null ? null : run(s); + }); + return fetcher; +} + +function fakeSleeper() { + const sleeps = []; + const sleeper = jest.fn(async (ms) => { + sleeps.push(ms); + }); + return { sleeper, sleeps }; +} + +describe('effective defaults', () => { + test('fall back to the documented defaults', () => { + expect(effectiveWaitTime(undefined)).toBe(POLL_OPTIONS.DEFAULT_WAIT_TIME_SECONDS); + expect(effectiveMaxAttempts(undefined)).toBe(POLL_OPTIONS.DEFAULT_MAX_ATTEMPTS); + expect(effectiveWaitTime({ waitTime: 5 })).toBe(5); + expect(effectiveMaxAttempts({ maxAttempts: 2 })).toBe(2); + }); +}); + +describe('pollRun', () => { + test('returns immediately on a terminal first lookup without sleeping', async () => { + for (const terminal of [DetectRunStatus.SUCCESS, DetectRunStatus.FAILED, DetectRunStatus.UNKNOWN]) { + const fetcher = sequence([terminal]); + const { sleeper, sleeps } = fakeSleeper(); + const out = await pollRun('run-1', {}, fetcher, undefined, sleeper); + expect(out.status).toBe(terminal); + expect(fetcher).toHaveBeenCalledTimes(1); + expect(sleeps).toEqual([]); + } + }); + + test('backs off 1, 2, 4, 8, 16, 32, 32 seconds between lookups until terminal', async () => { + const fetcher = sequence(['QUEUED', 'IN_PROGRESS', 'IN_PROGRESS', 'IN_PROGRESS', 'IN_PROGRESS', 'IN_PROGRESS', 'IN_PROGRESS', 'SUCCESS']); + const { sleeper, sleeps } = fakeSleeper(); + const out = await pollRun('run-1', { waitTime: 300, maxAttempts: 23 }, fetcher, undefined, sleeper); + expect(out.status).toBe(DetectRunStatus.SUCCESS); + expect(fetcher).toHaveBeenCalledTimes(8); + expect(sleeps).toEqual([1000, 2000, 4000, 8000, 16000, 32000, 32000]); + }); + + test('trims the last sleep so the total never exceeds waitTime, then returns the non-terminal run', async () => { + const fetcher = sequence(['IN_PROGRESS']); + const { sleeper, sleeps } = fakeSleeper(); + const out = await pollRun('run-1', { waitTime: 5, maxAttempts: 23 }, fetcher, undefined, sleeper); + expect(out.status).toBe(DetectRunStatus.IN_PROGRESS); + // 1 + 2 = 3 elapsed; next delay would be 4 but only 2 remain. + expect(sleeps).toEqual([1000, 2000, 2000]); + expect(fetcher).toHaveBeenCalledTimes(4); + }); + + test('stops after maxAttempts lookups', async () => { + const fetcher = sequence(['QUEUED']); + const { sleeper, sleeps } = fakeSleeper(); + const out = await pollRun('run-1', { waitTime: 300, maxAttempts: 3 }, fetcher, undefined, sleeper); + expect(out.status).toBe(DetectRunStatus.QUEUED); + expect(fetcher).toHaveBeenCalledTimes(3); + expect(sleeps).toEqual([1000, 2000]); + }); + + test('a fetcher that returns nothing is treated as an UNKNOWN terminal run', async () => { + const fetcher = sequence([null]); + const { sleeper } = fakeSleeper(); + const out = await pollRun('run-1', {}, fetcher, undefined, sleeper); + expect(out.status).toBe(DetectRunStatus.UNKNOWN); + expect(out.runId).toBe('run-1'); + }); + + test('propagates a fetcher failure', async () => { + const fetcher = jest.fn(async () => { + throw new Error('lookup failed'); + }); + const { sleeper } = fakeSleeper(); + await expect(pollRun('run-1', {}, fetcher, undefined, sleeper)).rejects.toThrow('lookup failed'); + }); +}); diff --git a/common/test/detect/test-types.js b/common/test/detect/test-types.js new file mode 100644 index 00000000..4f326727 --- /dev/null +++ b/common/test/detect/test-types.js @@ -0,0 +1,29 @@ +// Stand-ins for the vault-specific Detect types that each SDK (skyvault, flowvault) ships. The +// common layer only knows the contracts, so its tests use these the way a module would: a +// source enum per operation and a controller that resolves wire output types onto them. +import { BaseDetectController } from '../../src/detect/controller'; + +export const TestDeidentifySourceType = Object.freeze({ + BASE64: 'BASE64', + SKYFLOW_ID: 'SKYFLOW_ID', + PRESIGNED_URL: 'PRESIGNED_URL', +}); + +export const TestReidentifySourceType = Object.freeze({ + BASE64: 'BASE64', + PRESIGNED_URL: 'PRESIGNED_URL', +}); + +/** A module-style controller: the base class plus the two source resolvers every SDK provides. */ +export class TestDetectController extends BaseDetectController { + toDeidentifySource(value) { + return BaseDetectController.enumOrNull(TestDeidentifySourceType, value); + } + + toReidentifySource(value) { + return BaseDetectController.enumOrNull(TestReidentifySourceType, value); + } +} + +export const resolveDeidentifySource = (value) => BaseDetectController.enumOrNull(TestDeidentifySourceType, value); +export const resolveReidentifySource = (value) => BaseDetectController.enumOrNull(TestReidentifySourceType, value); diff --git a/common/test/detect/validations.test.js b/common/test/detect/validations.test.js new file mode 100644 index 00000000..c553a98e --- /dev/null +++ b/common/test/detect/validations.test.js @@ -0,0 +1,252 @@ +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { + validateDeidentifyStringRequest, + validateDeidentifyFileRequest, + validatePollOptions, + validateReidentifyStringRequest, + validateReidentifyFileRequest, + validateGetRunRequest, + validateCheckGuardrailsRequest, +} from '../../src/detect/internal/validations'; +import DeidentifyStringRequest from '../../src/detect/model/request/deidentify-string'; +import DeidentifyFileRequest from '../../src/detect/model/request/deidentify-file'; +import ReidentifyStringRequest from '../../src/detect/model/request/reidentify-string'; +import ReidentifyFileRequest from '../../src/detect/model/request/reidentify-file'; +import GetRunRequest from '../../src/detect/model/request/get-run'; +import CheckGuardrailsRequest from '../../src/detect/model/request/check-guardrails'; +import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType } from './test-types'; +import SkyflowError from '../../src/error'; +import SKYFLOW_ERROR_CODE from '../../src/error/codes'; + +beforeAll(() => { + jest.spyOn(console, 'error').mockImplementation(() => undefined); + jest.spyOn(console, 'log').mockImplementation(() => undefined); +}); +afterAll(() => jest.restoreAllMocks()); + +function expectCode(fn, code) { + let thrown; + try { + fn(); + } catch (e) { + thrown = e; + } + expect(thrown).toBeInstanceOf(SkyflowError); + expect(thrown.error.httpCode).toBe(code.httpCode); + expect(thrown.message).toBe(code.message); +} + +describe('validateDeidentifyStringRequest', () => { + test('rejects a missing request', () => { + expectCode(() => validateDeidentifyStringRequest(null), SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_STRING_REQUEST); + }); + test.each(['', ' ', undefined, 42])('rejects text %p', (text) => { + expectCode( + () => validateDeidentifyStringRequest(new DeidentifyStringRequest(text)), + SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_DEIDENTIFY_STRING, + ); + }); + test('accepts text with or without a configuration', () => { + expect(() => validateDeidentifyStringRequest(new DeidentifyStringRequest('hi'))).not.toThrow(); + expect(() => + validateDeidentifyStringRequest(new DeidentifyStringRequest('hi', { configurationId: 'cfg' })), + ).not.toThrow(); + expect(() => + validateDeidentifyStringRequest(new DeidentifyStringRequest('hi', { configuration: { detect: {} } })), + ).not.toThrow(); + }); +}); + +describe('validateDeidentifyFileRequest', () => { + let tmpDir; + let filePath; + beforeAll(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'detect-validations-')); + filePath = path.join(tmpDir, 'notes.txt'); + fs.writeFileSync(filePath, 'hello'); + }); + afterAll(() => fs.rmSync(tmpDir, { recursive: true, force: true })); + + const fileLike = () => ({ name: 'a.txt', arrayBuffer: async () => new ArrayBuffer(0) }); + + test('rejects a missing request', () => { + expectCode(() => validateDeidentifyFileRequest(undefined), SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); + }); + test('rejects more than one file source', () => { + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ file: fileLike(), filePath })), + SKYFLOW_ERROR_CODE.MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE, + ); + expectCode( + () => + validateDeidentifyFileRequest( + new DeidentifyFileRequest({ dataSource: DeidentifySourceType.BASE64, value: 'abc', filePath }), + ), + SKYFLOW_ERROR_CODE.MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE, + ); + }); + test('rejects a file that is not File-like', () => { + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ file: { name: 'x' } })), + SKYFLOW_ERROR_CODE.INVALID_FILE_IN_DEIDENTIFY_FILE, + ); + }); + test('rejects a filePath that does not exist or is a directory', () => { + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ filePath: path.join(tmpDir, 'missing.txt') })), + SKYFLOW_ERROR_CODE.FILE_NOT_FOUND_IN_DEIDENTIFY_FILE, + ); + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ filePath: tmpDir })), + SKYFLOW_ERROR_CODE.FILE_NOT_FOUND_IN_DEIDENTIFY_FILE, + ); + }); + test('requires exactly one of filePath, file or value', () => { + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({})), + SKYFLOW_ERROR_CODE.MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE, + ); + // dataSource alone is not a source: it only describes value. + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ dataSource: DeidentifySourceType.SKYFLOW_ID })), + SKYFLOW_ERROR_CODE.MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE, + ); + // An empty value still counts as the value source, so it collides with filePath. + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ value: '', filePath })), + SKYFLOW_ERROR_CODE.MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE, + ); + }); + test('requires dataSource and a non-empty value for a value-based request', () => { + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ value: 'abc' })), + SKYFLOW_ERROR_CODE.INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE, + ); + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ dataSource: DeidentifySourceType.BASE64, value: '' })), + SKYFLOW_ERROR_CODE.INVALID_VALUE_IN_DEIDENTIFY_FILE, + ); + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ dataSource: DeidentifySourceType.BASE64, value: ' ' })), + SKYFLOW_ERROR_CODE.INVALID_VALUE_IN_DEIDENTIFY_FILE, + ); + }); + test('accepts each single source', () => { + expect(() => validateDeidentifyFileRequest(new DeidentifyFileRequest({ file: fileLike() }))).not.toThrow(); + expect(() => validateDeidentifyFileRequest(new DeidentifyFileRequest({ filePath }))).not.toThrow(); + expect(() => + validateDeidentifyFileRequest( + new DeidentifyFileRequest({ dataSource: DeidentifySourceType.PRESIGNED_URL, value: 'https://x' }), + ), + ).not.toThrow(); + }); + test('rejects outputDirectory without pollOptions, accepts it with pollOptions', () => { + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ filePath, outputDirectory: '/tmp/out' })), + SKYFLOW_ERROR_CODE.OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS, + ); + expect(() => + validateDeidentifyFileRequest(new DeidentifyFileRequest({ filePath, outputDirectory: '/tmp/out', pollOptions: {} })), + ).not.toThrow(); + }); + test('validates poll options bounds', () => { + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ filePath, pollOptions: { waitTime: 0 } })), + SKYFLOW_ERROR_CODE.INVALID_WAIT_TIME_IN_POLL_OPTIONS, + ); + expectCode( + () => validateDeidentifyFileRequest(new DeidentifyFileRequest({ filePath, pollOptions: { maxAttempts: 24 } })), + SKYFLOW_ERROR_CODE.INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS, + ); + }); +}); + +describe('validatePollOptions', () => { + test('accepts undefined and in-range values', () => { + expect(() => validatePollOptions(undefined)).not.toThrow(); + expect(() => validatePollOptions({})).not.toThrow(); + expect(() => validatePollOptions({ waitTime: 1, maxAttempts: 1 })).not.toThrow(); + expect(() => validatePollOptions({ waitTime: 300, maxAttempts: 23 })).not.toThrow(); + }); + test.each([301, -1, 'x', NaN])('rejects waitTime %p', (waitTime) => { + expectCode(() => validatePollOptions({ waitTime }), SKYFLOW_ERROR_CODE.INVALID_WAIT_TIME_IN_POLL_OPTIONS); + }); + test.each([0, 24, 'x'])('rejects maxAttempts %p', (maxAttempts) => { + expectCode(() => validatePollOptions({ maxAttempts }), SKYFLOW_ERROR_CODE.INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS); + }); +}); + +describe('validateReidentifyStringRequest', () => { + test('rejects a missing request and blank text', () => { + expectCode(() => validateReidentifyStringRequest(null), SKYFLOW_ERROR_CODE.INVALID_REIDENTIFY_STRING_REQUEST); + expectCode( + () => validateReidentifyStringRequest(new ReidentifyStringRequest(' ')), + SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_REIDENTIFY_STRING, + ); + }); + test('rejects a non-array redactionLevel and accepts an array', () => { + expectCode( + () => validateReidentifyStringRequest(new ReidentifyStringRequest('t', { redactionLevel: {} })), + SKYFLOW_ERROR_CODE.INVALID_REDACTION_LEVEL, + ); + expect(() => + validateReidentifyStringRequest(new ReidentifyStringRequest('t', { redactionLevel: [{ tokenGroupName: 'g' }] })), + ).not.toThrow(); + }); +}); + +describe('validateReidentifyFileRequest', () => { + test('rejects a missing request, dataSource and value', () => { + expectCode(() => validateReidentifyFileRequest(undefined), SKYFLOW_ERROR_CODE.INVALID_REIDENTIFY_FILE_REQUEST); + expectCode( + () => validateReidentifyFileRequest(new ReidentifyFileRequest(undefined, 'abc')), + SKYFLOW_ERROR_CODE.INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE, + ); + expectCode( + () => validateReidentifyFileRequest(new ReidentifyFileRequest(ReidentifySourceType.BASE64, '')), + SKYFLOW_ERROR_CODE.INVALID_VALUE_IN_REIDENTIFY_FILE, + ); + }); + test('accepts a valid request', () => { + expect(() => + validateReidentifyFileRequest(new ReidentifyFileRequest(ReidentifySourceType.PRESIGNED_URL, 'https://x')), + ).not.toThrow(); + }); +}); + +describe('validateGetRunRequest', () => { + test('rejects a missing request and blank runId', () => { + expectCode(() => validateGetRunRequest(null), SKYFLOW_ERROR_CODE.INVALID_GET_RUN_REQUEST); + expectCode(() => validateGetRunRequest(new GetRunRequest('')), SKYFLOW_ERROR_CODE.INVALID_RUN_ID_IN_GET_RUN); + expectCode(() => validateGetRunRequest(new GetRunRequest(undefined)), SKYFLOW_ERROR_CODE.INVALID_RUN_ID_IN_GET_RUN); + }); + test('accepts a run id', () => { + expect(() => validateGetRunRequest(new GetRunRequest('run-1'))).not.toThrow(); + }); +}); + +describe('validateCheckGuardrailsRequest', () => { + test('rejects a missing request, blank text and bad denyTopics', () => { + expectCode(() => validateCheckGuardrailsRequest(null), SKYFLOW_ERROR_CODE.INVALID_CHECK_GUARDRAILS_REQUEST); + expectCode( + () => validateCheckGuardrailsRequest(new CheckGuardrailsRequest('')), + SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_CHECK_GUARDRAILS, + ); + expectCode( + () => validateCheckGuardrailsRequest(new CheckGuardrailsRequest('t', { denyTopics: 'politics' })), + SKYFLOW_ERROR_CODE.INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS, + ); + expectCode( + () => validateCheckGuardrailsRequest(new CheckGuardrailsRequest('t', { denyTopics: ['ok', 3] })), + SKYFLOW_ERROR_CODE.INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS, + ); + }); + test('accepts text with optional checks', () => { + expect(() => validateCheckGuardrailsRequest(new CheckGuardrailsRequest('t'))).not.toThrow(); + expect(() => + validateCheckGuardrailsRequest(new CheckGuardrailsRequest('t', { checkToxicity: true, denyTopics: ['a'] })), + ).not.toThrow(); + }); +}); diff --git a/common/test/service-account/token.test.js b/common/test/service-account/token.test.js index 9d73568f..60e4d581 100644 --- a/common/test/service-account/token.test.js +++ b/common/test/service-account/token.test.js @@ -27,7 +27,7 @@ jest.mock('../../src/service-account/client', () => { __esModule: true, default: jest.fn().mockImplementation(() => ({ authApi: { - authenticationServiceGetAuthToken: jest.fn(() => ({ + getBearerToken: jest.fn(() => ({ withRawResponse: jest.fn().mockResolvedValueOnce({ data: { accessToken: 'mocked_access_token', @@ -375,7 +375,7 @@ describe('getToken Tests', () => { beforeEach(() => { mockClient = { authApi: { - authenticationServiceGetAuthToken: jest.fn(), + getBearerToken: jest.fn(), }, }; jest.spyOn(jwt, 'sign').mockReturnValue('mocked_token'); @@ -396,7 +396,7 @@ describe('getToken Tests', () => { test("should get Bearer Token with valid credentials", async () => { const filePath = 'test/demo-credentials/valid.json'; - mockClient.authApi.authenticationServiceGetAuthToken.mockImplementation(() => ({ + mockClient.authApi.getBearerToken.mockImplementation(() => ({ withRawResponse: jest.fn().mockResolvedValueOnce({ data: mockTokenResponse, rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } } @@ -506,7 +506,7 @@ describe('getToken and getSignedTokens tokenUri override tests', () => { const Client = jest.requireMock('../../src/service-account/client').default; Client.mockImplementationOnce(() => ({ authApi: { - authenticationServiceGetAuthToken: jest.fn(() => ({ + getBearerToken: jest.fn(() => ({ withRawResponse: jest.fn().mockRejectedValueOnce(new Error('API rejection')) })) } @@ -524,7 +524,7 @@ describe('getToken and getSignedTokens tokenUri override tests', () => { const Client = jest.requireMock('../../src/service-account/client').default; Client.mockImplementationOnce(() => ({ authApi: { - authenticationServiceGetAuthToken: jest.fn(() => ({ + getBearerToken: jest.fn(() => ({ withRawResponse: jest.fn().mockResolvedValueOnce({ data: { accessToken: 'mocked_access_token', tokenType: 'Bearer' }, rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } } @@ -546,7 +546,7 @@ describe('getToken and getSignedTokens tokenUri override tests', () => { const Client = jest.requireMock('../../src/service-account/client').default; Client.mockImplementationOnce(() => ({ authApi: { - authenticationServiceGetAuthToken: jest.fn(() => ({ + getBearerToken: jest.fn(() => ({ withRawResponse: jest.fn().mockResolvedValueOnce({ data: { accessToken: 'mocked_access_token', tokenType: 'Bearer' }, rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } } @@ -730,7 +730,7 @@ describe('deprecated BearerTokenOptions.roleIDs normalization', () => { const Client = jest.requireMock('../../src/service-account/client').default; Client.mockImplementationOnce(() => ({ authApi: { - authenticationServiceGetAuthToken: jest.fn(() => ({ + getBearerToken: jest.fn(() => ({ withRawResponse: jest.fn().mockResolvedValueOnce({ data: { accessToken: 'mocked_access_token', tokenType: 'Bearer' }, rawResponse: { @@ -759,7 +759,7 @@ describe('deprecated BearerTokenOptions.roleIDs normalization', () => { const Client = jest.requireMock('../../src/service-account/client').default; Client.mockImplementationOnce(() => ({ authApi: { - authenticationServiceGetAuthToken: jest.fn(() => ({ + getBearerToken: jest.fn(() => ({ withRawResponse: jest.fn().mockResolvedValueOnce({ data: { accessToken: 'mocked_access_token', tokenType: 'Bearer' }, rawResponse: { @@ -790,7 +790,7 @@ describe('deprecated BearerTokenOptions.roleIDs normalization', () => { const Client = jest.requireMock('../../src/service-account/client').default; Client.mockImplementationOnce(() => ({ authApi: { - authenticationServiceGetAuthToken: jest.fn(() => ({ + getBearerToken: jest.fn(() => ({ withRawResponse: jest.fn().mockResolvedValueOnce({ data: { accessToken: 'mocked_access_token', tokenType: 'Bearer' }, rawResponse: { diff --git a/docs/api_reference.md b/docs/api_reference.md index 92ac7ad3..7a4d2d12 100644 --- a/docs/api_reference.md +++ b/docs/api_reference.md @@ -1,6 +1,6 @@ # API Reference -A reference for the public Skyflow Node SDK surface: client-management methods, request classes, options classes, response classes, enums, Detect helper classes, and service-account functions. For task-oriented usage and examples, see the [README](../README.md). +A reference for the public Skyflow Node SDK surface: client-management methods, request classes, options classes, response classes, enums, the Detect API (shared with the flowvault SDK), and service-account functions. For task-oriented usage and examples, see the [README](../README.md). All properties, setters, and enum values below are taken directly from the SDK source. @@ -11,7 +11,7 @@ All properties, setters, and enum values below are taken directly from the SDK s - [Options classes](#options-classes) - [Response classes](#response-classes) - [Enums](#enums) -- [Detect helper classes](#detect-helper-classes) +- [Detect API](#detect-api) - [Service account functions](#service-account-functions) --- @@ -23,7 +23,7 @@ The `Skyflow` client exposes the following methods to manage configuration at ru | Method | Description | |--------|-------------| | `vault(vaultId?)` | Return a `VaultController` for the given vault ID. Defaults to the first configured vault. | -| `detect(vaultId?)` | Return a `DetectController` for the given vault ID. | +| `detect(vaultId?)` | Return a `DetectController` for the given vault ID. See [Detect API](#detect-api). | | `connection(connectionId?)` | Return a `ConnectionController` for the given connection ID. | | `addVaultConfig(config)` | Add a vault configuration after initialization. Throws if the vault ID already exists. | | `updateVaultConfig(config)` | Update an existing vault configuration. | @@ -141,52 +141,6 @@ Passed to `vault().uploadFile()`. Use [`FileUploadOptions`](#fileuploadoptions) | `table` | `string` | Target table name. | | `columnName` | `string` | File column name. | -### `DeidentifyTextRequest` - -Passed to `detect().deidentifyText()`. - -| Constructor argument | Type | Description | -|---|---|---| -| `text` | `string` | Text to de-identify. | - -Configure additional behavior with [`DeidentifyTextOptions`](#deidentifytextoptions). - -### `ReidentifyTextRequest` - -Passed to `detect().reidentifyText()`. - -| Constructor argument | Type | Description | -|---|---|---| -| `text` | `string` | The redacted/de-identified text to re-identify. | - -Configure additional behavior with [`ReidentifyTextOptions`](#reidentifytextoptions). - -### `FileInput` - -Passed as the constructor argument to `DeidentifyFileRequest`. Provide **exactly one** source. - -| Variant | Shape | Description | -|---|---|---| -| `FileObject` | `{ file: File }` | An in-memory `File` object. | -| `Filepath` | `{ filePath: string }` | Path to a file on disk. | - -### `DeidentifyFileRequest` - -Passed to `detect().deidentifyFile()`. Provide either a `File` object or a file path — not both. - -| Constructor argument | Type | Description | -|---|---|---| -| `fileInput` | [`FileInput`](#fileinput) | `{ file: File }` or `{ filePath: string }`. | - -Configure additional behavior with [`DeidentifyFileOptions`](#deidentifyfileoptions). - -### `GetDetectRunRequest` - -Passed to `detect().getDetectRun()`. - -| Constructor argument | Type | Description | -|---|---|---| -| `options` | `{ runId: string }` | The `runId` returned by a prior `deidentifyFile` call. | ### `InvokeConnectionRequest` @@ -258,43 +212,6 @@ Provide **exactly one** file source: `setFileObject`, `setFilePath`, or `setBase | `setBase64(data)` | `string` | Base64-encoded file content. Use with `setFileName`. | | `setFileName(name)` | `string` | Filename for base64 uploads (e.g. `"document.pdf"`). | -### `DeidentifyTextOptions` - -| Setter | Type | Description | -|---|---|---| -| `setEntities(entities)` | `DetectEntities[]` | Entity types to detect and de-identify. | -| `setAllowRegexList(patterns)` | `string[]` | Regex patterns that always match as entities. | -| `setRestrictRegexList(patterns)` | `string[]` | Regex patterns that never match as entities. | -| `setTokenFormat(format)` | `TokenFormat` | Token format per entity type. See [`TokenFormat`](#tokenformat). | -| `setTransformations(t)` | `Transformations` | Custom transformations. See [`Transformations`](#transformations). | - -### `ReidentifyTextOptions` - -| Setter | Type | Description | -|---|---|---| -| `setRedactedEntities(entities)` | `DetectEntities[]` | Entity types to keep redacted. | -| `setMaskedEntities(entities)` | `DetectEntities[]` | Entity types to mask. | -| `setPlainTextEntities(entities)` | `DetectEntities[]` | Entity types to reveal as plain text. | - -### `DeidentifyFileOptions` - -| Setter | Type | Description | -|---|---|---| -| `setEntities(entities)` | `DetectEntities[]` | Entity types to detect. | -| `setAllowRegexList(patterns)` | `string[]` | Regex patterns to always treat as detectable. | -| `setRestrictRegexList(patterns)` | `string[]` | Regex patterns to exclude from detection. | -| `setTokenFormat(format)` | `TokenFormat` | Token format per entity type. | -| `setTransformations(t)` | `Transformations` | Custom transformations (not supported for Documents, Images, or PDFs). | -| `setOutputDirectory(path)` | `string` | Directory to write the processed file. Not supported in Cloudflare Workers. | -| `setWaitTime(seconds)` | `number` | Max seconds to poll (≤ 64). Returns `runId` + `status` if exceeded. | -| `setOutputProcessedImage(bool)` | `boolean` | Include the processed image in the response. Images only. | -| `setOutputOcrText(bool)` | `boolean` | Include OCR-extracted text. Images only. | -| `setMaskingMethod(method)` | `MaskingMethod` | Masking method for image entities. See [`MaskingMethod`](#maskingmethod). | -| `setPixelDensity(dpi)` | `number` | DPI for image rendering. Images/PDFs only. | -| `setMaxResolution(px)` | `number` | Maximum pixel dimension of the output. Images/PDFs only. | -| `setOutputProcessedAudio(bool)` | `boolean` | Include processed audio in the response. Audio only. | -| `setOutputTranscription(format)` | `DetectOutputTranscription` | Transcription format. See [`DetectOutputTranscription`](#detectoutputtranscription). Audio only. | -| `setBleep(bleep)` | `Bleep` | Bleep tone config for redacted audio spans. See [`Bleep`](#bleep). Audio only. | --- @@ -386,46 +303,6 @@ Returned by `connection().invoke()`. | `metadata` | `{ requestId: string } \| undefined` | Response metadata including the server-side `requestId`. | | `errors` | `SkyflowRecordError[] \| null` | See note above. | -### `DeidentifyTextResponse` - -Returned by `detect().deidentifyText()`. - -| Field | Type | Description | -|---|---|---| -| `processedText` | `string` | The de-identified text. | -| `entities` | `Array<{token, value, textIndex: IndexRange, processedIndex: IndexRange, entity, scores}>` | Detected entities. | -| `wordCount` | `number` | Word count of the input text. | -| `charCount` | `number` | Character count of the input text. | -| `errors` | `SkyflowRecordError[] \| null` | See note above. | - -### `ReidentifyTextResponse` - -Returned by `detect().reidentifyText()`. - -| Field | Type | Description | -|---|---|---| -| `processedText` | `string` | The re-identified text. | - -### `DeidentifyFileResponse` - -Returned by `detect().deidentifyFile()` and `detect().getDetectRun()`. All fields are optional — populated based on file type and processing status. When processing exceeds `setWaitTime`, only `runId` and `status` are set; poll with `getDetectRun`. - -| Field | Type | Description | -|---|---|---| -| `fileBase64` | `string \| undefined` | The processed file as a base64 string. | -| `file` | `File \| undefined` | The processed `File` object. | -| `type` | `string \| undefined` | MIME type of the processed file. | -| `extension` | `string \| undefined` | File extension of the processed file. | -| `wordCount` | `number \| undefined` | Word count (text-bearing files). | -| `charCount` | `number \| undefined` | Character count (text-bearing files). | -| `sizeInKb` | `number \| undefined` | Size of the processed file in KB. | -| `durationInSeconds` | `number \| undefined` | Duration in seconds (audio files). | -| `pageCount` | `number \| undefined` | Page count (PDF/document files). | -| `slideCount` | `number \| undefined` | Slide count (presentation files). | -| `entities` | `Array<{file: string; extension: string}>` | Detected entity files. Defaults to `[]`. | -| `runId` | `string \| undefined` | Run identifier; pass to `getDetectRun` to poll. | -| `status` | `string \| undefined` | Processing status (`IN_PROGRESS`, `SUCCESS`, `FAILED`). | -| `errors` | `SkyflowRecordError[] \| null` | See note above. | ### `SkyflowRecordError` @@ -481,15 +358,6 @@ BYOT mode for `InsertOptions` and `UpdateOptions`. | `TokenMode.ENABLE` | Caller-supplied tokens accepted as-is. | | `TokenMode.ENABLE_STRICT` | Caller-supplied tokens validated before accepting. | -### `TokenType` - -Token format used in `TokenFormat.setDefault()`. - -| Value | Description | -|---|---| -| `TokenType.VAULT_TOKEN` | Standard vault token. | -| `TokenType.ENTITY_UNIQUE_COUNTER` | Entity-unique counter token. | -| `TokenType.ENTITY_ONLY` | Entity label only (no token value). | ### `OrderByEnum` @@ -507,66 +375,306 @@ HTTP method for `InvokeConnectionRequest`. Values: `GET`, `POST`, `PUT`, `PATCH`. -### `MaskingMethod` -Image masking method for `DeidentifyFileOptions.setMaskingMethod()`. +--- -Values: `Blackbox`, `Blur`. +## Detect API -### `DetectOutputTranscription` +Shared by the `skyflow-node` and `skyflow-flowvault-node` SDKs: every class, interface and enum below is importable from either package. `skyflowClient.detect()` and `skyflowClient.detect(vaultId)` return a `DetectController`. Usage and examples: [skyvault README](../README.md#detect), [flowvault README](../flowvault/README.md#detect). -Audio transcription format for `DeidentifyFileOptions.setOutputTranscription()`. +| Method | Request | Response | +|--------|---------|----------| +| `deidentifyString` | [`DeidentifyStringRequest`](#deidentifystringrequest) | [`DeidentifyStringResponse`](#deidentifystringresponse) | +| `deidentifyFile` | [`DeidentifyFileRequest`](#deidentifyfilerequest) | [`DeidentifyFileResponse`](#deidentifyfileresponse) | +| `reidentifyString` | [`ReidentifyStringRequest`](#reidentifystringrequest) | [`ReidentifyStringResponse`](#reidentifystringresponse) | +| `reidentifyFile` | [`ReidentifyFileRequest`](#reidentifyfilerequest) | [`ReidentifyFileResponse`](#reidentifyfileresponse) | +| `getRun` | [`GetRunRequest`](#getrunrequest) | [`GetRunResponse`](#getrunresponse) | +| `checkGuardrails` | [`CheckGuardrailsRequest`](#checkguardrailsrequest) | [`CheckGuardrailsResponse`](#checkguardrailsresponse) | -Values: `DIARIZED_TRANSCRIPTION`, `MEDICAL_DIARIZED_TRANSCRIPTION`, `MEDICAL_TRANSCRIPTION`, `TRANSCRIPTION`, `PLAINTEXT_TRANSCRIPTION`. +Configuration and helper types: [`DetectConfiguration`](#detectconfiguration), [`Detect`](#detect), [`Entity`](#entity), [`Transformation` and `ShiftDates`](#transformation-and-shiftdates), [`CustomType`](#customtype), [`ObjectEntity`](#objectentity), [`FileMapping`](#filemapping), [`Media`](#media) (with `Audio`, `Bleep`, `Document`, `Pdf`, `Image`), [`PollOptions`](#polloptions), [`RedactionLevel`](#redactionlevel), [`DetectedEntity`](#detectedentity), [`EntityLocation`](#entitylocation), [`Metrics`](#metrics), [`FileOutput`](#fileoutput). Enums: [Detect enums](#detect-enums). -### `DetectEntities` +Request classes take their required fields as constructor arguments and their optional fields in a trailing object; every field is also exposed as a settable property. Response classes are read-only data containers. -Entity types Skyflow Detect can identify — e.g. `SSN`, `CREDIT_CARD`, `NAME`, `DOB`, `PHONE_NUMBER`, `EMAIL`, `ACCOUNT_NUMBER`, and more. Import from `'skyflow-node'`. +### `DeidentifyStringRequest` ---- +`new DeidentifyStringRequest(text, fields?)` — passed to `detect().deidentifyString()`. -## Detect helper classes +| Field | Type | Description | +|---|---|---| +| `text` | `string` | **Required.** Text to de-identify. | +| `configurationId` | `string` | Id of a Detect configuration created through the management API. Sending it together with `configuration` is rejected by the API. | +| `configuration` | [`DetectConfiguration`](#detectconfiguration) | Inline configuration. With neither this nor `configurationId` set, the SDK sends only the vault id and the server applies the vault's defaults. | -All importable from `'skyflow-node'`. +### `DeidentifyFileRequest` -### `TokenFormat` +`new DeidentifyFileRequest(fields)` — passed to `detect().deidentifyFile()`. Supply the file through exactly one of `value` (with a `dataSource`), `file` or `filePath`. -Controls the token representation per entity type in de-identification. Pass to `DeidentifyTextOptions.setTokenFormat()` or `DeidentifyFileOptions.setTokenFormat()`. +| Field | Type | Description | +|---|---|---| +| `dataSource` | [`DeidentifySourceType`](#detect-enums) | How `value` supplies the file, as your SDK's enum: `BASE64` or `PRESIGNED_URL`; the skyvault SDK also offers `SKYFLOW_ID`. | +| `value` | `string` | Base64 content, a skyflow id, or a presigned URL, per `dataSource`. | +| `file` | `File` | In-memory `File`; read and base64-encoded when sent, `dataSource` set to `BASE64`. Requires Node.js v20+. | +| `filePath` | `string` | Path to a local file; same handling as `file`. | +| `dataFormat` | [`FileDataFormat`](#detect-enums) | File format. Inferred from the file name's extension when omitted for a local file. | +| `configurationId` | `string` | Id of a stored Detect configuration. Sending it together with `configuration` is rejected by the API. | +| `configuration` | [`DetectConfiguration`](#detectconfiguration) | Inline configuration. With neither this nor `configurationId` set, the server applies the vault's defaults. | +| `pollOptions` | [`PollOptions`](#polloptions) | Poll the run after submitting. Omit to submit only and return the run id. | +| `outputDirectory` | `string` | Where outputs are written on success. Requires `pollOptions` and a local file; setting it without `pollOptions` fails client-side validation. Defaults to the working directory. | -| Setter | Type | Description | +### `ReidentifyStringRequest` + +`new ReidentifyStringRequest(text, fields?)` — passed to `detect().reidentifyString()`. + +| Field | Type | Description | |---|---|---| -| `setDefault(type)` | `TokenType` | Default token format for all detected entities. | -| `setVaultToken(entities)` | `DetectEntities[]` | Use vault tokens for these specific entity types. | -| `setEntityUniqueCounter(entities)` | `DetectEntities[]` | Use entity-unique counter tokens for these types. | -| `setEntityOnly(entities)` | `DetectEntities[]` | Use entity-only (no token) for these types. | +| `text` | `string` | **Required.** The de-identified text to re-identify. | +| `redactionLevel` | [`RedactionLevel[]`](#redactionlevel) | Per-entity redaction rules. Without rules every token is replaced by its original value. | -### `Transformations` +### `ReidentifyFileRequest` -Specifies custom transformations applied to detected entities during de-identification. Pass to `DeidentifyTextOptions.setTransformations()`. +`new ReidentifyFileRequest(dataSource, value, fields?)` — passed to `detect().reidentifyFile()`. Synchronous: the re-identified file comes back in the response. -| Setter | Type | Description | +| Field | Type | Description | |---|---|---| -| `setShiftDays(config)` | `{ max: number; min: number; entities: DetectEntities[] }` | Shift date entities by a random number of days within `[min, max]`. | +| `dataSource` | [`ReidentifySourceType`](#detect-enums) | **Required.** Your SDK's enum: `BASE64` or `PRESIGNED_URL`. Re-identification does not accept a skyflow id as the source. | +| `value` | `string` | **Required.** Base64 content or a presigned URL, per `dataSource`. | +| `dataFormat` | [`FileDataFormat`](#detect-enums) | Limited to the formats that support `VAULT_TOKEN`: `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` or `PPTX`. | +| `redactionLevel` | [`RedactionLevel[]`](#redactionlevel) | Redaction rules. | -### `Bleep` +### `GetRunRequest` -Audio bleep tone configuration. Pass to `DeidentifyFileOptions.setBleep()`. +`new GetRunRequest(runId)` — passed to `detect().getRun()`. -| Setter | Type | Description | +| Field | Type | Description | +|---|---|---| +| `runId` | `string` | **Required.** The run id returned by a prior `deidentifyFile` call. | + +### `CheckGuardrailsRequest` + +`new CheckGuardrailsRequest(text, fields?)` — passed to `detect().checkGuardrails()`. + +| Field | Type | Description | +|---|---|---| +| `text` | `string` | **Required.** Text to screen. | +| `checkToxicity` | `boolean` | Whether to check for toxicity. Sent only when set. | +| `denyTopics` | `string[]` | Topics to deny. Sent only when set. | + +### `DeidentifyStringResponse` + +Returned by `detect().deidentifyString()`. + +| Field | Type | Description | +|---|---|---| +| `processedText` | `string \| null` | The de-identified text with entity values replaced by tokens. | +| `entities` | [`DetectedEntity[]`](#detectedentity) | Detected entities, never `null`. | +| `metrics` | [`Metrics`](#metrics)` \| null` | Word and character counts. | + +### `ReidentifyStringResponse` + +Returned by `detect().reidentifyString()`. + +| Field | Type | Description | +|---|---|---| +| `processedText` | `string \| null` | The re-identified text. | +| `metrics` | [`Metrics`](#metrics)` \| null` | Word and character counts. | + +### `DeidentifyFileResponse` + +Returned by `detect().deidentifyFile()`. Without `pollOptions` only `runId` is set. + +| Field | Type | Description | +|---|---|---| +| `runId` | `string \| null` | Run id, always present once the run was submitted. | +| `status` | [`DetectRunStatus`](#detect-enums)` \| null` | Run status when polled, `null` when submit only. | +| `outputType` | [`DeidentifySourceType`](#detect-enums)` \| null` | The run's output type resolved onto your SDK's enum; `null` until the run completes or when the value is not one the SDK offers. | +| `output` | [`FileOutput[]`](#fileoutput) | Every output of the run, empty until complete, never `null`. | +| `metrics` | [`Metrics`](#metrics)` \| null` | Size, counts, pages, slides, duration. `null` until complete. | +| `message` | `string \| null` | Failure detail when the status is `FAILED`. | + +### `ReidentifyFileResponse` + +Returned by `detect().reidentifyFile()`. + +| Field | Type | Description | +|---|---|---| +| `status` | [`DetectRunStatus`](#detect-enums) | Status of the re-identification. | +| `outputType` | [`ReidentifySourceType`](#detect-enums)` \| null` | The output type resolved onto your SDK's enum; `null` when the value is not one the SDK offers. | +| `output` | [`FileOutput[]`](#fileoutput) | Re-identified file outputs, never `null`. | +| `metrics` | [`Metrics`](#metrics)` \| null` | Size and counts. | + +### `GetRunResponse` + +Returned by `detect().getRun()`. Same fields as [`DeidentifyFileResponse`](#deidentifyfileresponse). + +### `CheckGuardrailsResponse` + +Returned by `detect().checkGuardrails()`. + +| Field | Type | Description | +|---|---|---| +| `text` | `string \| null` | The screened text. | +| `validation` | [`GuardrailsValidation`](#detect-enums) | `PASSED` or `FAILED`; `UNKNOWN` for a value this SDK version does not know. | +| `toxic` | `boolean \| null` | Whether the text was judged toxic, `null` when the check did not run. | +| `deniedTopic` | `boolean \| null` | Whether a denied topic was found, `null` when no topics were supplied. | + +### `DetectConfiguration` + +Inline Detect policy for `deidentifyString` and `deidentifyFile`. A plain interface: write it as an object literal. + +| Field | Type | Description | +|---|---|---| +| `name` | `string` | Configuration name. | +| `description` | `string` | Configuration description. | +| `detect` | [`Detect`](#detect) | What to detect and how to de-identify it. | +| `fileMapping` | [`FileMapping[]`](#filemapping) | Per-column handling for structured files. File inputs only. | +| `media` | [`Media`](#media) | Audio, document and image options for file calls. File inputs only. | + +### `Detect` + +| Field | Type | Description | +|---|---|---| +| `entities` | [`Entity[]`](#entity) | Entities to detect, each with its own de-identification type. | +| `objectEntities` | [`ObjectEntity[]`](#objectentity) | Visual objects to detect in images. | +| `restrict` | `string[]` | Regexes to always treat as sensitive. | +| `skip` | `string[]` | Regexes to never treat as sensitive. | +| `customTypes` | [`CustomType[]`](#customtype) | Regex-defined entity types. | +| `returnEntities` | `ReturnEntitiesType` | Whether raw values come back in the response. Server default `EXCLUDE_SENSITIVE_DATA`. | + +### `Entity` + +| Field | Type | Description | +|---|---|---| +| `entityType` | `EntityType` | Entity type. Server default `ALL`. | +| `deidentificationType` | `DeidentificationType` | How the value is replaced. Server default `ENTITY_UNIQUE_COUNTER`. | +| `destination` | `string` | For `VAULT_TOKEN`: where the value is stored. A `table.column` on skyvault vaults, a token group name on flowvault vaults. Must already exist. | +| `transformation` | [`Transformation`](#transformation-and-shiftdates) | Date shifting. | + +### `Transformation` and `ShiftDates` + +`{ shiftDates?: { minDays?: number; maxDays?: number } }`. Dates are shifted by a random amount within the range. + +### `CustomType` + +| Field | Type | Description | +|---|---|---| +| `label` | `string` | Becomes the entity type on the response. | +| `match` | `string[]` | Regexes that define the type. | + +### `ObjectEntity` + +| Field | Type | Description | +|---|---|---| +| `entityType` | `ObjectEntityType` | `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`. | +| `deidentificationType` | `ObjectDeidentificationType` | `REDACT` or `UNREDACT`. | + +### `FileMapping` + +| Field | Description | +|---|---| +| `source` | Source column. | +| `destination` | Destination column. | +| `outputFileNameSuffix` | Suffix added to the output file name. | +| `entitiesDestination` | Column that receives the detected entities. | +| `objectEntitiesDestination` | Column that receives the detected object entities. | + +### `Media` + +`{ audio?: Audio; document?: Document; image?: Image }`. + +| Interface | Fields | +|-------|-----------------| +| `Audio` | `outputProcessedAudio?: boolean`, `outputTranscription?: OutputTranscriptionType`, `bleep?: Bleep` | +| `Bleep` | `startPadding?: number`, `stopPadding?: number`, `frequency?: number`, `gain?: number` | +| `Document` | `pdf?: Pdf` | +| `Pdf` | `processingMode?: PdfProcessingMode`, `density?: number`, `maxResolution?: number` | +| `Image` | `outputProcessedImage?: boolean`, `outputOcrText?: boolean`, `maskingMethod?: MaskingMethod` | + +### `PollOptions` + +Opts `deidentifyFile` into submit-and-poll mode. The SDK polls with exponential backoff (1, 2, 4, 8, 16, then 32 seconds). The limits are exported as `POLL_OPTIONS`. + +| Field | Default | Description | +|---|---|---| +| `waitTime` | `60` | Maximum total seconds to keep polling, 1 to 300. | +| `maxAttempts` | `23` | Maximum number of run lookups, 1 to 23. | + +### `RedactionLevel` + +Element of `redactionLevel` on `ReidentifyStringRequest` and `ReidentifyFileRequest`. Each SDK ships its own `RedactionLevel` with the pair its vault understands; the shared request models accept it through the `RedactionRule` contract in `skyflow-common`. Every field is optional on the API; the server rejects invalid combinations. + +skyvault SDK (`skyflow-node`): + +| Field | Type | Description | +|---|---|---| +| `entityName` | `EntityType` | Entity type the rule applies to. | +| `redactionType` | [`RedactionType`](#redactiontype) | `PLAIN_TEXT`, `MASKED`, `REDACTED` or `DEFAULT`. | + +flowvault SDK (`skyflow-flowvault-node`): + +| Field | Type | Description | +|---|---|---| +| `tokenGroupName` | `string` | Token group the rule applies to. | +| `redactionPattern` | `string` | Name of a redaction pattern configured in the vault. | + +### `DetectedEntity` + +Element of `DeidentifyStringResponse.entities`. + +| Field | Type | Description | +|---|---|---| +| `token` | `string \| null` | The token that replaced the value, e.g. `NAME_1`. | +| `value` | `string \| null` | The original value; empty unless `returnEntities` is `ALL`. | +| `location` | [`EntityLocation`](#entitylocation)` \| null` | Offsets in the original and processed text. | +| `entityType` | `string \| null` | Entity type label, including custom type labels. | +| `entityScores` | `Record` | Confidence scores per entity type. | + +### `EntityLocation` + +| Field | Type | Description | +|---|---|---| +| `startIndex` / `endIndex` | `number \| null` | Offsets into the original text. | +| `startIndexProcessed` / `endIndexProcessed` | `number \| null` | Offsets into the processed text. | + +### `Metrics` + +| Field | Type | Description | |---|---|---| -| `setGain(value)` | `number` | Volume of the bleep tone (0.0 – 1.0). | -| `setFrequency(hz)` | `number` | Frequency in hertz. | -| `setStartPadding(seconds)` | `number` | Silence padding before the bleep. | -| `setStopPadding(seconds)` | `number` | Silence padding after the bleep. | +| `size` | `number \| null` | Size in kilobytes. | +| `wordCount` | `number \| null` | Word count. | +| `characterCount` | `number \| null` | Character count. | +| `pages` | `number \| null` | Page count, documents only. | +| `slides` | `number \| null` | Slide count, presentations only. | +| `duration` | `number \| null` | Duration in seconds, audio only. | -### `IndexRange` +### `FileOutput` -Character position range within text. Used in `DeidentifyTextResponse.entities[].textIndex` and `.processedIndex`. +Element of `output` on the file responses. | Field | Type | Description | |---|---|---| -| `start` | `number \| undefined` | Start offset. | -| `end` | `number \| undefined` | End offset. | +| `processedFile` | `string \| null` | Base64 content or a presigned URL, per the response's `outputType`. | +| `processedFileType` | `string \| null` | `REDACTED_FILE`, `ENTITIES`, `OBJECT_ENTITIES`, `REDACTED_TEXT`, and so on. | +| `processedFileExtension` | `FileDataFormat \| null` | Extension of the output. | + +### Detect enums + +All string enums whose values match the wire values. + +| Enum | Values | +|------|--------| +| `EntityType` | the 70 entity types (`NAME`, `EMAIL_ADDRESS`, `SSN`, `CREDIT_CARD`, `DOB`, ...), plus `ALL` and `NONE` | +| `DeidentificationType` | `ENTITY_UNIQUE_COUNTER` _(server default)_, `ENTITY_ONLY`, `VAULT_TOKEN`, `MASK`, `UNKNOWN` | +| `DeidentifySourceType` | Per SDK: skyvault `BASE64`, `SKYFLOW_ID`, `PRESIGNED_URL`; flowvault `BASE64`, `PRESIGNED_URL`. Input source for `deidentifyFile`, and the `outputType` of `DeidentifyFileResponse` and `GetRunResponse`. | +| `ReidentifySourceType` | Per SDK, `BASE64`, `PRESIGNED_URL` on both. Input source for `reidentifyFile` and the `outputType` of `ReidentifyFileResponse`. | +| `FileDataFormat` | `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML`, `ZIP` | +| `DetectRunStatus` | `QUEUED`, `IN_PROGRESS`, `SUCCESS`, `FAILED`, `UNKNOWN`. `SUCCESS`, `FAILED` and `UNKNOWN` end polling. | +| `ReturnEntitiesType` | `EXCLUDE_SENSITIVE_DATA` _(server default)_, `ALL`, `NONE` | +| `MaskingMethod` | `NONE`, `BLUR`, `BLACKBOX` | +| `OutputTranscriptionType` | `NONE`, `TRANSCRIPTION`, `MEDICAL_TRANSCRIPTION`, `DIARIZED_TRANSCRIPTION`, `MEDICAL_DIARIZED_TRANSCRIPTION` | +| `PdfProcessingMode` | `NONE`, `OCR`, `TEXT_LAYER` | +| `ObjectEntityType` | `NONE`, `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE`, `ALL` | +| `ObjectDeidentificationType` | `UNKNOWN`, `REDACT`, `UNREDACT` | +| `GuardrailsValidation` | `PASSED`, `FAILED`, `UNKNOWN` | --- diff --git a/docs/sdk-split-plan.md b/docs/sdk-split-plan.md index 20b3584d..71b7767e 100644 --- a/docs/sdk-split-plan.md +++ b/docs/sdk-split-plan.md @@ -41,6 +41,13 @@ Everything that is not tied to a specific wire contract: `skyflow-node`. - `base/` — `BaseOptions` + `BaseRequest` / `BaseResponse` marker classes (shared extension points). +- `detect/` — the **Detect V2** surface shared by both SDKs (mirrors Java's `com.skyflow.detect`): + request/response models, the inline-configuration types and enums, client-side validation, + request/response mappers, the run poller, the output writer, and **`BaseDetectController`**, + which drives the generated `stringsV2` / `filesV2` / `guardrailsV2` clients. Each layer's + `DetectController` extends it and supplies only its `VaultClient` (URL, vault id, token + resolution via `resolveAuthToken()`, log level, timeout/retry overrides). The v1 detect + interfaces that used to live in skyvault were removed when this landed. - `skyflow/base.ts` — **`BaseSkyflow`**, the shared facade base (vault-client registry, common credentials, log level, and the `addVaultConfig`/`updateVaultConfig`/`getVaultConfig`/ `removeVaultConfig`/`setLogLevel`/`updateLogLevel`/`getLogLevel`/`updateSkyflowCredentials`/ @@ -49,8 +56,9 @@ Everything that is not tied to a specific wire contract: `common` `BaseSkyflow`. ### `skyvault` (→ `skyflow-node`) -The original contract, unchanged behavior: vault data plane (insert/get/update/delete/ -detokenize/tokenize/query/file-upload) + connections + detect. Its `Skyflow` **extends +The original contract: vault data plane (insert/get/update/delete/ +detokenize/tokenize/query/file-upload) + connections + Detect V2 (the shared controller from +`common`). Its `Skyflow` **extends `BaseSkyflow`** (adding the detect controller per vault + the connection registry/`connection()`); its `VaultClient`, controllers and models import shared code from `skyflow-common`; its concrete model classes extend the `common` `Base*` classes; the per-request validators live here. It keeps @@ -58,9 +66,10 @@ its existing data-plane generated client under `skyvault/src/ _generated_`. ### `flowvault` (→ `skyflow-flowvault-node`) The new v2 contract (unary only): insert/get/update/delete/query/detokenize/getTokens/uploadFiles/ -deleteFiles. Its `Skyflow` **extends `BaseSkyflow`** (just `buildVaultEntry` + `vault()`); its -models/options extend the same `Base*` classes; it wires to its own generated client at -`flowvault/_generated_/rest/src` (`.records`/`.tokens`/`.files`/`.query`). No connections/detect. +deleteFiles, plus Detect V2 through the shared controller. Its `Skyflow` **extends `BaseSkyflow`** +(`buildVaultEntry` + `vault()` + `detect()`); its models/options extend the same `Base*` classes; +it wires to its own generated client at `flowvault/_generated_/rest/src` +(`.records`/`.tokens`/`.files`/`.query`). No connections. ## Consumer experience (one import per SDK) @@ -137,7 +146,8 @@ runtime value) and the full type surface is preserved in the `.d.ts`; marking th ## Generated code (maintainer-owned) The OpenAPI spec is split and the client/transport code is generated **per module**: -`common/src/_generated_/` (auth), `skyvault/src/ _generated_` (v1 data plane), +`common/src/_generated_/` (auth, token exchange, and the shared Detect v2 `stringsV2` / +`filesV2` / `guardrailsV2` clients), `skyvault/src/ _generated_` (v1 data plane), `flowvault/_generated_/rest/src` (v2 data plane). Generated directories are treated as **provided inputs** — not generated, moved, renamed, or hand-edited by the split. Where the hand-written layer needs something the generated surface doesn't provide, it is **flagged** for the maintainer diff --git a/flowvault/README.md b/flowvault/README.md index e63671e4..62c0817e 100644 --- a/flowvault/README.md +++ b/flowvault/README.md @@ -2,7 +2,8 @@ The FlowVault (v2) Node.js SDK for Skyflow's data privacy vault — a unary, strongly-typed client for the v2 vault data plane. With it you can securely insert, retrieve, update, delete, query, -detokenize, tokenize, and attach files to records in a Flow DB vault. +detokenize, tokenize, and attach files to records in a Flow DB vault, and de-identify and +re-identify sensitive data in text and files with Skyflow Detect. > This is a **separate product** from `skyflow-node` (the SkyVault SDK). Install exactly one: > `npm i skyflow-flowvault-node` **or** `npm i skyflow-node`. A lower `flowvault` version number @@ -16,6 +17,7 @@ detokenize, tokenize, and attach files to records in a Flow DB vault. - [Authenticate](#authenticate) - [Initialize the client](#initialize-the-client) - [Vault operations](#vault-operations) +- [Detect](#detect) - [Custom request headers](#custom-request-headers) - [Error handling](#error-handling) - [Samples](#samples) @@ -27,7 +29,10 @@ detokenize, tokenize, and attach files to records in a Flow DB vault. or async variants. - **Unified responses.** Each response carries a single `records` list, with per-record `httpCode`, `error`, and `requestId`. Whole-call failures throw a `SkyflowError`. -- **Multi-vault.** One client can hold several vault configs; pick one per call with `vault(vaultId)`. +- **Multi-vault.** One client can hold several vault configs; pick one per call with `vault(vaultId)` + or `detect(vaultId)`. +- **Detect.** The same Detect V2 API as the SkyVault SDK, scoped to a Flow DB vault — see + [Detect](#detect). ## Install @@ -259,6 +264,291 @@ const req = new DeleteFilesRequest([ const res = await vault.deleteFiles(req); ``` +## Detect + +Skyflow Detect finds and de-identifies sensitive data in free text and files, then re-identifies it on demand. The SDK exposes the Detect V2 API through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault, and the controller honours the same credentials, [timeouts and retries](#timeouts-and-retries) as `vault()`. + +Detect configurations are created in Studio or through the management API. The SDK does not create, update or delete them. + +| Method | Request | Response | Description | +|--------|---------|----------|-------------| +| `deidentifyString(request)` | [`DeidentifyStringRequest`](../docs/api_reference.md#deidentifystringrequest) | [`DeidentifyStringResponse`](../docs/api_reference.md#deidentifystringresponse) | Detect and replace sensitive data in a string | +| `deidentifyFile(request)` | [`DeidentifyFileRequest`](../docs/api_reference.md#deidentifyfilerequest) | [`DeidentifyFileResponse`](../docs/api_reference.md#deidentifyfileresponse) | Submit a file for de-identification, optionally polling the run to completion | +| `reidentifyString(request)` | [`ReidentifyStringRequest`](../docs/api_reference.md#reidentifystringrequest) | [`ReidentifyStringResponse`](../docs/api_reference.md#reidentifystringresponse) | Replace tokens in a string with their original or redacted values | +| `reidentifyFile(request)` | [`ReidentifyFileRequest`](../docs/api_reference.md#reidentifyfilerequest) | [`ReidentifyFileResponse`](../docs/api_reference.md#reidentifyfileresponse) | Replace tokens in a file with their original or redacted values | +| `getRun(request)` | [`GetRunRequest`](../docs/api_reference.md#getrunrequest) | [`GetRunResponse`](../docs/api_reference.md#getrunresponse) | Retrieve the status and outputs of a Detect file run | +| `checkGuardrails(request)` | [`CheckGuardrailsRequest`](../docs/api_reference.md#checkguardrailsrequest) | [`CheckGuardrailsResponse`](../docs/api_reference.md#checkguardrailsresponse) | Screen text for toxicity and denied topics | + +**Before you start** + +- **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy that grants the Detokenization action on the destination, unless it is a Vault Owner. +- **Configuration.** `configurationId` is the id of a Detect configuration created through the management API. The alternative is an inline `DetectConfiguration` passed as `configuration` (see [Detect configuration](#detect-configuration)). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings. Sending both is rejected by the API with HTTP 400. +- **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the token group named by `destination`. That token group must already exist in the vault; Detect writes into it but does not create it. +- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](../docs/api_reference.md#detect-api). + +### De-identify a string: `.deidentifyString(request)` + +`DeidentifyStringRequest` takes the text plus, optionally, `configurationId` (a stored Detect configuration) or `configuration` (an inline `DetectConfiguration`). The SDK fills the vault id into the configuration from the controller. + +```typescript +import { + DeidentifyStringRequest, + DeidentifyStringResponse, + DetectConfiguration, + EntityType, + DeidentificationType, + ReturnEntitiesType, + SkyflowError, +} from 'skyflow-flowvault-node'; + +// Stored configuration +const stored = new DeidentifyStringRequest( + 'My name is John Doe, and my email is johndoe@acme.com.', + { configurationId: '' }, +); +const response: DeidentifyStringResponse = await skyflowClient.detect().deidentifyString(stored); +console.log(response.processedText); +// My name is [NAME_1], and my email is [EMAIL_ADDRESS_1]. + +// Inline configuration +const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '', // token group that stores the value + }, + { entityType: EntityType.ALL, deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER }, + ], + skip: ['Skyflow'], + returnEntities: ReturnEntitiesType.EXCLUDE_SENSITIVE_DATA, + }, +}; +const inline = new DeidentifyStringRequest( + 'My name is John Doe, and my email is johndoe@acme.com.', + { configuration }, +); +const inlineResponse = await skyflowClient.detect().deidentifyString(inline); +for (const entity of inlineResponse.entities) { + console.log(entity.entityType, entity.token, entity.location?.startIndex, entity.location?.endIndex); +} +console.log(inlineResponse.metrics?.wordCount); +``` + +`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. + +> [!TIP] +> See the full example in the samples directory: [deidentify-string.ts](samples/detect-api/deidentify-string.ts) + +### Detect configuration + +`DetectConfiguration` is the inline form of a stored configuration and accepts the same options. `detect` applies to string and file inputs; `fileMapping` and `media` apply to file inputs only. All of these are plain TypeScript interfaces, so write them as object literals. + +| Type | Fields | Notes | +|------|--------|-------| +| `Detect` | `entities`, `objectEntities`, `restrict`, `skip`, `customTypes`, `returnEntities` | `restrict` and `skip` are regexes that are always, respectively never, treated as sensitive. `returnEntities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. | +| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` is used only with `VAULT_TOKEN`. | +| `Transformation` / `ShiftDates` | `shiftDates`; `minDays`, `maxDays` | Shifts detected dates by a random number of days within the range. | +| `CustomType` | `label`, `match` | A regex-defined entity type. `label` is reported as the entity type of its matches. | +| `ObjectEntity` | `entityType`, `deidentificationType` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. | +| `FileMapping` | `source`, `destination`, `outputFileNameSuffix`, `entitiesDestination`, `objectEntitiesDestination` | Where the processed file and the entity files are written. File inputs only. | +| `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. | +| `Audio` / `Bleep` | `outputProcessedAudio`, `outputTranscription`, `bleep`; `startPadding`, `stopPadding`, `frequency`, `gain` | `outputTranscription` is an `OutputTranscriptionType`. `Bleep` is the tone laid over detected speech; paddings are in seconds, frequency in Hz. | +| `Document` / `Pdf` | `pdf`; `processingMode`, `density`, `maxResolution` | `processingMode` is a `PdfProcessingMode` (`NONE`, `OCR` or `TEXT_LAYER`). `density` is the rendering DPI. | +| `Image` | `outputProcessedImage`, `outputOcrText`, `maskingMethod` | `maskingMethod` is a `MaskingMethod` (`NONE`, `BLUR` or `BLACKBOX`). | + +```typescript +import { + DetectConfiguration, + EntityType, + DeidentificationType, + ObjectEntityType, + ObjectDeidentificationType, + MaskingMethod, + PdfProcessingMode, + OutputTranscriptionType, +} from 'skyflow-flowvault-node'; + +const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.DOB, + deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, + transformation: { shiftDates: { minDays: 10, maxDays: 30 } }, + }, + { entityType: EntityType.ALL }, + ], + objectEntities: [{ entityType: ObjectEntityType.FACE, deidentificationType: ObjectDeidentificationType.REDACT }], + customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], + restrict: ['[0-9]{3}-[0-9]{2}-[0-9]{4}'], + }, + media: { + image: { maskingMethod: MaskingMethod.BLUR, outputOcrText: true }, + document: { pdf: { processingMode: PdfProcessingMode.OCR } }, + audio: { + outputTranscription: OutputTranscriptionType.TRANSCRIPTION, + bleep: { startPadding: 0.5, stopPadding: 0.5 }, + }, + }, +}; +``` + +### De-identify a file: `.deidentifyFile(request)` + +File de-identification is asynchronous on the API: the submit call returns a run id and the result is retrieved from the run. `deidentifyFile` exposes both modes. + +- **Submit only** (default): one call, returns `runId`. `status` is `null` and nothing is written to disk. Retrieve the result later with [`getRun`](#get-run-getrunrequest). +- **Submit and poll**: set `pollOptions`. The SDK polls the run with exponential backoff (1, 2, 4, 8, 16, then 32 seconds between lookups) until it is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `waitTime` (default 60 s, max 300) or `maxAttempts` (default 23, max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED` so you can resume with `getRun`. + +The file is supplied in one of three ways: a `dataSource` plus `value` (`DeidentifySourceType.BASE64` content or a `DeidentifySourceType.PRESIGNED_URL`; flowvault vaults do not take a skyflow id as a source), a `file` (a web `File`, Node.js 20+), or a `filePath`. A local file is read, base64-encoded, and its `dataFormat` inferred from the extension. Supply exactly one of `value`, `file` or `filePath`. `configurationId` and `configuration` follow the same rules as for strings. + +`dataFormat` accepts every `FileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. + +`outputDirectory` is meaningful only when polling. Setting it without `pollOptions` fails client-side validation before any request is sent. + +```typescript +import { DeidentifyFileRequest, DeidentifyFileResponse, DetectRunStatus } from 'skyflow-flowvault-node'; + +// Submit only +const submitted: DeidentifyFileResponse = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ + filePath: '/data/intake/patient-notes.pdf', // sent as BASE64, format inferred as PDF + configurationId: '', + }), +); +const runId = submitted.runId; + +// Submit and poll +const response = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ + filePath: '/data/intake/patient-notes.pdf', + configurationId: '', + pollOptions: { waitTime: 120 }, + outputDirectory: '/data/processed', // optional + }), +); + +switch (response.status) { + case DetectRunStatus.SUCCESS: + // For local-file requests the outputs are already on disk: + // /data/processed/processed-patient-notes.pdf the redacted file + // /data/processed/processed-entities-patient-notes.json the detected entities + // response.output and response.metrics carry the same data. + break; + case DetectRunStatus.FAILED: + console.error(response.message); + break; + default: // IN_PROGRESS or QUEUED: budget ran out, resume with getRun(runId) + break; +} +``` + +Outputs are written only when polling reached `SUCCESS`, the request was built from a local file, and the run delivered them as base64. A request built from a `PRESIGNED_URL` has no local name to derive from, and presigned-URL outputs are not fetched, so in those cases read `output` and write the files yourself. Files are named `processed-.`, with the entities output as `processed-entities-.json`; `outputDirectory` is created if missing and defaults to the working directory. Writing outputs needs the Node.js file system and is not supported in Cloudflare Workers. + +> [!TIP] +> See the full examples in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](samples/detect-api/deidentify-file-async.ts) + +### Re-identify a string: `.reidentifyString(request)` + +Re-identification looks tokens up in the vault, so it applies to values that were de-identified with `VAULT_TOKEN`. Placeholders produced by `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not stored anywhere and cannot be re-identified. + +`ReidentifyStringRequest` takes the tokenized text and an optional list of `RedactionLevel` rules. For flowvault vaults each rule targets a `tokenGroupName` and applies a named `redactionPattern` configured in the vault. Without rules every token is replaced by its original value. + +```typescript +import { ReidentifyStringRequest, ReidentifyStringResponse } from 'skyflow-flowvault-node'; + +const response: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest('My name is [NAME_1].', { + redactionLevel: [{ tokenGroupName: '', redactionPattern: '' }], + }), +); +console.log(response.processedText); +// My name is J**n D*e. +``` + +`RedactionLevel` is the flowvault SDK's own type and has only `tokenGroupName` and `redactionPattern`; the `entityName` and `redactionType` rules of the skyvault SDK do not exist here. Both fields are optional on the API, so the SDK passes the list through as given and the server rejects invalid combinations. + +> [!TIP] +> See the full example in the samples directory: [reidentify-string.ts](samples/detect-api/reidentify-string.ts) + +### Re-identify a file: `.reidentifyFile(request)` + +`reidentifyFile` is synchronous: the re-identified file comes back in the response, not through a run id. `ReidentifyFileRequest` takes a `dataSource` (`ReidentifySourceType.BASE64` or `PRESIGNED_URL`), the matching `value`, an optional `dataFormat` and the same optional `RedactionLevel` rules as `reidentifyString`. `dataFormat` is limited to the formats that support `VAULT_TOKEN`: `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. + +```typescript +import * as fs from 'fs'; +import { ReidentifyFileRequest, ReidentifySourceType, FileDataFormat, DetectRunStatus } from 'skyflow-flowvault-node'; + +const content = fs.readFileSync('tokenized.txt').toString('base64'); + +const response = await skyflowClient.detect().reidentifyFile( + new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: FileDataFormat.TXT }), +); + +if (response.status === DetectRunStatus.SUCCESS) { + for (const output of response.output) { + // output.processedFile is base64 or a presigned URL, per response.outputType + const processed = Buffer.from(output.processedFile ?? '', 'base64'); + } +} +``` + +`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. + +> [!TIP] +> See the full example in the samples directory: [reidentify-file.ts](samples/detect-api/reidentify-file.ts) + +### Get run: `.getRun(request)` + +File de-identification is asynchronous: the submit call returns a run id, and `getRun` retrieves the run's status and, once it has succeeded, its outputs. `output` is empty and `outputType` and `metrics` are `null` until the run completes. + +```typescript +import { GetRunRequest, GetRunResponse, DetectRunStatus } from 'skyflow-flowvault-node'; + +const run: GetRunResponse = await skyflowClient.detect().getRun(new GetRunRequest('')); + +switch (run.status) { + case DetectRunStatus.SUCCESS: + for (const output of run.output) { + // output.processedFile is base64 or a presigned URL, per run.outputType + // output.processedFileType is e.g. REDACTED_FILE or ENTITIES + // output.processedFileExtension is a FileDataFormat such as PDF + } + break; + case DetectRunStatus.FAILED: + console.error(run.message); + break; + default: // QUEUED, IN_PROGRESS, UNKNOWN: poll again later + break; +} +``` + +> [!TIP] +> See the full example in the samples directory: [get-run.ts](samples/detect-api/get-run.ts) + +### Check guardrails: `.checkGuardrails(request)` + +`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. + +```typescript +import { CheckGuardrailsRequest, GuardrailsValidation } from 'skyflow-flowvault-node'; + +const response = await skyflowClient.detect().checkGuardrails( + new CheckGuardrailsRequest(userPrompt, { checkToxicity: true, denyTopics: ['politics', 'medical advice'] }), +); + +if (response.validation === GuardrailsValidation.FAILED) { + throw new Error(`blocked: toxic=${response.toxic} deniedTopic=${response.deniedTopic}`); +} +``` + +All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling). Client-side validation fails before any network call. + +> [!TIP] +> See the full example in the samples directory: [check-guardrails.ts](samples/detect-api/check-guardrails.ts) + ## Custom request headers The SDK supports a fixed set of custom headers, listed on the `CustomHeaderKey` diff --git a/flowvault/api-report/skyflow-flowvault-node.api.md b/flowvault/api-report/skyflow-flowvault-node.api.md index 4d356193..0fbee5f2 100644 --- a/flowvault/api-report/skyflow-flowvault-node.api.md +++ b/flowvault/api-report/skyflow-flowvault-node.api.md @@ -9,6 +9,7 @@ import { BaseDeleteFilesRequest } from 'skyflow-common'; import { BaseDeleteFilesResponse } from 'skyflow-common'; import { BaseDeleteRequest } from 'skyflow-common'; import { BaseDeleteResponse } from 'skyflow-common'; +import { BaseDetectController } from 'skyflow-common'; import { BaseDetokenizeRequest } from 'skyflow-common'; import { BaseDetokenizeResponse } from 'skyflow-common'; import { BaseGetRequest } from 'skyflow-common'; @@ -449,6 +450,10 @@ class Skyflow extends BaseSkyflow { constructor(config: SkyflowConfig); // (undocumented) protected buildVaultEntry(config: VaultConfig): ClientEntry; + // Warning: (ae-forgotten-export) The symbol "DetectController" needs to be exported by the entry point index.d.ts + // + // (undocumented) + detect(vaultId?: string): DetectController; // Warning: (ae-forgotten-export) The symbol "VaultController" needs to be exported by the entry point index.d.ts // // (undocumented) diff --git a/flowvault/samples/README.md b/flowvault/samples/README.md index 685930dd..582b4ac9 100644 --- a/flowvault/samples/README.md +++ b/flowvault/samples/README.md @@ -50,6 +50,18 @@ npx ts-node flowvault/samples/vault-api/insert-records.ts | [upload-files.ts](vault-api/upload-files.ts) | Upload files to file columns (`filePath` / `base64` / `file`) | | [delete-files.ts](vault-api/delete-files.ts) | Delete files from file columns (by `skyflowId` or `uniqueValues`) | +## Detect + +| Sample | Demonstrates | +|---|---| +| [deidentify-string.ts](detect-api/deidentify-string.ts) | De-identify text with a stored or inline Detect configuration | +| [deidentify-file.ts](detect-api/deidentify-file.ts) | Submit a file (local path, skyflow id or presigned URL) and get a run id | +| [deidentify-file-async.ts](detect-api/deidentify-file-async.ts) | Submit a file and let the SDK poll the run, writing outputs to disk | +| [reidentify-string.ts](detect-api/reidentify-string.ts) | Restore original values from de-identified text, with redaction rules | +| [reidentify-file.ts](detect-api/reidentify-file.ts) | Restore original values in a tokenized file | +| [get-run.ts](detect-api/get-run.ts) | Check the status and outputs of a file de-identification run | +| [check-guardrails.ts](detect-api/check-guardrails.ts) | Screen text for toxicity and denied topics | + ## Custom headers & HTTP config | Sample | Demonstrates | diff --git a/flowvault/samples/detect-api/check-guardrails.ts b/flowvault/samples/detect-api/check-guardrails.ts new file mode 100644 index 00000000..19d38dd3 --- /dev/null +++ b/flowvault/samples/detect-api/check-guardrails.ts @@ -0,0 +1,82 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + CheckGuardrailsRequest, + CheckGuardrailsResponse, + GuardrailsValidation, +} from 'skyflow-flowvault-node'; + +/** + * Skyflow Detect: check guardrails + * + * checkGuardrails screens text for toxicity and for a list of denied topics, which makes it a + * natural pre-LLM check. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Screen a prompt for toxicity and denied topics + * 3. Block the prompt when the validation fails + * 4. Handle errors + */ +async function performCheckGuardrails() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Screen the text. checkToxicity and denyTopics are optional and sent only when set. + const userPrompt = 'Tell me which candidate to vote for.'; + const request = new CheckGuardrailsRequest(userPrompt, { + checkToxicity: true, + denyTopics: ['politics', 'medical advice'], + }); + const response: CheckGuardrailsResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .checkGuardrails(request); + + // Step 5: Act on the result. toxic and deniedTopic are null when that check did not run. + console.log('Validation:', response.validation); + console.log('Toxic:', response.toxic, 'Denied topic:', response.deniedTopic); + if (response.validation === GuardrailsValidation.FAILED) { + console.log('Prompt blocked before reaching the model.'); + } else { + console.log('Prompt allowed.'); + } + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the check guardrails function +performCheckGuardrails(); diff --git a/flowvault/samples/detect-api/deidentify-file-async.ts b/flowvault/samples/detect-api/deidentify-file-async.ts new file mode 100644 index 00000000..ae9bb9df --- /dev/null +++ b/flowvault/samples/detect-api/deidentify-file-async.ts @@ -0,0 +1,126 @@ +import * as fs from 'fs'; +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + DeidentifyFileRequest, + DeidentifyFileResponse, + DetectRunStatus, + DetectConfiguration, + EntityType, + DeidentificationType, + MaskingMethod, + PdfProcessingMode, +} from 'skyflow-flowvault-node'; + +/** + * Skyflow Detect: de-identify a file and poll the run to completion + * + * This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Submit a local file with pollOptions so the SDK polls the run for you + * 3. Write the processed outputs to an output directory on success + * 4. Submit an in-memory File object (Node.js 20+) with an inline configuration + * 5. Handle a run that is still in progress when the polling budget runs out + */ +async function performDeidentifyFileAndPoll() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Submit and poll. The SDK polls with exponential backoff (1, 2, 4, 8, 16, then + // 32 seconds between lookups) until the run is SUCCESS, FAILED or UNKNOWN, or until + // waitTime (default 60 s, max 300) or maxAttempts (default 23) is used up. + const request = new DeidentifyFileRequest({ + filePath: '', // e.g. /data/intake/patient-notes.pdf + configurationId: '', + pollOptions: { waitTime: 120, maxAttempts: 10 }, + outputDirectory: '', // optional; created if missing, default: working directory + }); + const response: DeidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(request); + + handleRun(response); + + // Step 5 (alternative): an in-memory File with an inline configuration + const buffer = fs.readFileSync(''); + const file = new File([new Uint8Array(buffer)], 'patient-notes.pdf'); + const configuration: DetectConfiguration = { + detect: { + entities: [ + { entityType: EntityType.NAME, deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER }, + { entityType: EntityType.ALL }, + ], + }, + media: { + image: { maskingMethod: MaskingMethod.BLUR, outputOcrText: true }, + document: { pdf: { processingMode: PdfProcessingMode.OCR } }, + }, + }; + const fileObjectResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(new DeidentifyFileRequest({ file, configuration, pollOptions: {} })); + + handleRun(fileObjectResponse); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +function handleRun(response: DeidentifyFileResponse) { + switch (response.status) { + case DetectRunStatus.SUCCESS: + // For local-file requests the outputs are already on disk as + // processed-. and processed-entities-.json. + console.log('Run succeeded:', response.runId); + for (const output of response.output) { + console.log(`${output.processedFileType} (${output.processedFileExtension})`); + } + console.log('Metrics:', response.metrics); + break; + case DetectRunStatus.FAILED: + console.error('Run failed:', response.message); + break; + default: + // IN_PROGRESS or QUEUED: the polling budget ran out. Resume with getRun(runId). + console.log(`Run ${response.runId} is still ${response.status}; poll it with getRun.`); + break; + } +} + +// Invoke the de-identify and poll function +performDeidentifyFileAndPoll(); diff --git a/flowvault/samples/detect-api/deidentify-file.ts b/flowvault/samples/detect-api/deidentify-file.ts new file mode 100644 index 00000000..8c010e30 --- /dev/null +++ b/flowvault/samples/detect-api/deidentify-file.ts @@ -0,0 +1,92 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + DeidentifyFileRequest, + DeidentifyFileResponse, + DeidentifySourceType, + FileDataFormat, +} from 'skyflow-flowvault-node'; + +/** + * Skyflow Detect: de-identify a file (submit only) + * + * File de-identification is asynchronous on the API. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Submit a local file for de-identification and receive a run id + * 3. Submit a file by skyflow id or presigned URL instead of a local file + * 4. Handle errors + * + * Retrieve the result later with getRun (see get-run.ts), or let the SDK poll for you + * (see deidentify-file-async.ts). + */ +async function performDeidentifyFile() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Submit a local file. It is read, base64-encoded and sent as BASE64; + // dataFormat is inferred from the extension unless you set it. + const localFileRequest = new DeidentifyFileRequest({ + filePath: '', // e.g. /data/intake/patient-notes.pdf + configurationId: '', // or configuration: { ... } for an inline configuration + }); + const submitted: DeidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(localFileRequest); + + console.log('Run id:', submitted.runId); // status, output and metrics are null until polled + // Pass submitted.runId to getRun (see get-run.ts) to retrieve the result. + + // Step 5 (alternative): submit a file at a presigned URL. Flowvault vaults take BASE64 or + // PRESIGNED_URL only; the flowvault DeidentifySourceType has no SKYFLOW_ID. + const remoteFileRequest = new DeidentifyFileRequest({ + dataSource: DeidentifySourceType.PRESIGNED_URL, + value: '', + dataFormat: FileDataFormat.PDF, + configurationId: '', + }); + const remoteSubmitted = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(remoteFileRequest); + console.log('Run id (remote file):', remoteSubmitted.runId); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the de-identify file function +performDeidentifyFile(); diff --git a/flowvault/samples/detect-api/deidentify-string.ts b/flowvault/samples/detect-api/deidentify-string.ts new file mode 100644 index 00000000..9c3d15f0 --- /dev/null +++ b/flowvault/samples/detect-api/deidentify-string.ts @@ -0,0 +1,109 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + DeidentifyStringRequest, + DeidentifyStringResponse, + DetectConfiguration, + EntityType, + DeidentificationType, + ReturnEntitiesType, +} from 'skyflow-flowvault-node'; + +/** + * Skyflow Detect: de-identify a string + * + * This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. De-identify text with a stored Detect configuration + * 3. De-identify text with an inline Detect configuration + * 4. Read the processed text, detected entities and metrics + * 5. Handle errors + */ +async function performDeidentifyString() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + const text = 'My name is John Doe, and my email is johndoe@acme.com.'; + + // Step 4: De-identify with a stored configuration (created through the management API) + const storedRequest = new DeidentifyStringRequest(text, { + configurationId: '', + }); + const storedResponse: DeidentifyStringResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyString(storedRequest); + console.log('Processed text (stored configuration):', storedResponse.processedText); + // My name is [NAME_1], and my email is [EMAIL_ADDRESS_1]. + + // Step 5: De-identify with an inline configuration + const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '', // token group that stores the value; must exist + }, + { + entityType: EntityType.ALL, + deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, + }, + ], + skip: ['Skyflow'], // never treat these matches as sensitive + returnEntities: ReturnEntitiesType.EXCLUDE_SENSITIVE_DATA, + }, + }; + const inlineRequest = new DeidentifyStringRequest(text, { configuration }); + const inlineResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyString(inlineRequest); + + console.log('Processed text (inline configuration):', inlineResponse.processedText); + for (const entity of inlineResponse.entities) { + console.log( + `${entity.entityType} -> ${entity.token} [${entity.location?.startIndex}, ${entity.location?.endIndex}]`, + ); + } + console.log('Word count:', inlineResponse.metrics?.wordCount); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the de-identify string function +performDeidentifyString(); diff --git a/flowvault/samples/detect-api/get-run.ts b/flowvault/samples/detect-api/get-run.ts new file mode 100644 index 00000000..9357731b --- /dev/null +++ b/flowvault/samples/detect-api/get-run.ts @@ -0,0 +1,90 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + GetRunRequest, + GetRunResponse, + DetectRunStatus, +} from 'skyflow-flowvault-node'; + +/** + * Skyflow Detect: get a file run + * + * File de-identification is asynchronous: deidentifyFile returns a run id, and getRun retrieves + * the run's status and, once it has succeeded, its outputs. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Look up a run by id + * 3. Act on the run status and read its outputs + * 4. Handle errors + */ +async function performGetRun() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Look up the run returned by a prior deidentifyFile call + const request = new GetRunRequest(''); + const run: GetRunResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .getRun(request); + + // Step 5: Act on the status + switch (run.status) { + case DetectRunStatus.SUCCESS: + for (const output of run.output) { + // output.processedFile is base64 or a presigned URL, per run.outputType + // output.processedFileType is e.g. REDACTED_FILE or ENTITIES + // output.processedFileExtension is a FileDataFormat such as PDF + console.log(`${output.processedFileType} (${output.processedFileExtension})`); + } + console.log('Output type:', run.outputType); + console.log('Metrics:', run.metrics); + break; + case DetectRunStatus.FAILED: + console.error('Run failed:', run.message); + break; + default: + // QUEUED, IN_PROGRESS, UNKNOWN: poll again later + console.log(`Run ${run.runId} is ${run.status}; try again later.`); + break; + } + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the get run function +performGetRun(); diff --git a/flowvault/samples/detect-api/reidentify-file.ts b/flowvault/samples/detect-api/reidentify-file.ts new file mode 100644 index 00000000..9b518506 --- /dev/null +++ b/flowvault/samples/detect-api/reidentify-file.ts @@ -0,0 +1,93 @@ +import * as fs from 'fs'; +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + ReidentifyFileRequest, + ReidentifyFileResponse, + ReidentifySourceType, + FileDataFormat, + DetectRunStatus, +} from 'skyflow-flowvault-node'; + +/** + * Skyflow Detect: re-identify a file + * + * reidentifyFile is synchronous: the re-identified file comes back in the response. This + * example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Send a tokenized local file as base64 with optional redaction rules + * 3. Decode the re-identified output + * 4. Handle errors + */ +async function performReidentifyFile() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Read the tokenized file and send it as BASE64. dataFormat is limited to the + // formats that support VAULT_TOKEN: TXT, CSV, JSON, JSONL, XML, DOCX, XLSX and PPTX. + const content = fs.readFileSync('').toString('base64'); + const request = new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { + dataFormat: FileDataFormat.TXT, + redactionLevel: [ + { tokenGroupName: '', redactionPattern: '' }, + ], + }); + const response: ReidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyFile(request); + + // Step 5: Decode the output. processedFile is base64 or a presigned URL, per outputType. + if (response.status === DetectRunStatus.SUCCESS) { + for (const output of response.output) { + if (output.processedFile) { + const bytes = Buffer.from(output.processedFile, 'base64'); + fs.writeFileSync(`reidentified.${output.processedFileExtension?.toLowerCase()}`, bytes); + console.log(`Wrote ${output.processedFileType} (${bytes.length} bytes)`); + } + } + console.log('Metrics:', response.metrics); + } else { + console.error('Re-identification did not succeed:', response.status); + } + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the re-identify file function +performReidentifyFile(); diff --git a/flowvault/samples/detect-api/reidentify-string.ts b/flowvault/samples/detect-api/reidentify-string.ts new file mode 100644 index 00000000..34233d43 --- /dev/null +++ b/flowvault/samples/detect-api/reidentify-string.ts @@ -0,0 +1,84 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + ReidentifyStringRequest, + ReidentifyStringResponse, +} from 'skyflow-flowvault-node'; + +/** + * Skyflow Detect: re-identify a string + * + * Re-identification looks tokens up in the vault, so it applies to values that were + * de-identified with VAULT_TOKEN. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Re-identify tokenized text, restoring the original values + * 3. Apply redaction rules so some entities come back masked or redacted + * 4. Handle errors + */ +async function performReidentifyString() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + const tokenizedText = 'My name is [NAME_1], and my SSN is [SSN_1].'; + + // Step 4: Without rules every token is replaced by its original value + const plainRequest = new ReidentifyStringRequest(tokenizedText); + const plainResponse: ReidentifyStringResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyString(plainRequest); + console.log('Re-identified text:', plainResponse.processedText); + + // Step 5: Redaction rules. Each rule targets a token group and applies a redaction pattern configured in the vault. + const redactedRequest = new ReidentifyStringRequest(tokenizedText, { + redactionLevel: [ + { tokenGroupName: '', redactionPattern: '' }, + { tokenGroupName: '', redactionPattern: '' }, + ], + }); + const redactedResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyString(redactedRequest); + console.log('Re-identified text with redaction rules:', redactedResponse.processedText); + console.log('Character count:', redactedResponse.metrics?.characterCount); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the re-identify string function +performReidentifyString(); diff --git a/flowvault/src/client/index.ts b/flowvault/src/client/index.ts index 62fa2c8b..45d577ef 100644 --- a/flowvault/src/client/index.ts +++ b/flowvault/src/client/index.ts @@ -137,6 +137,15 @@ class VaultClient { this.currentToken = authInfo.key; } + /** + * Resolve the bearer token or API key for the next call and return it. Used + * by the shared Detect controller, which builds its own generated clients. + */ + async resolveAuthToken(): Promise { + await this.getToken(); + return this.currentToken; + } + private supplierOptions(): Records.Options { return { baseUrl: () => this.url, diff --git a/flowvault/src/controller/detect/index.ts b/flowvault/src/controller/detect/index.ts new file mode 100644 index 00000000..4d19dd38 --- /dev/null +++ b/flowvault/src/controller/detect/index.ts @@ -0,0 +1,33 @@ +import { BaseDetectController } from 'skyflow-common'; +import VaultClient from '../../client'; +import { DeidentifySourceType, ReidentifySourceType } from '../../detect/types'; + +/** + * Skyflow Detect operations for a flowvault vault, returned by `Skyflow.detect()` + * and `Skyflow.detect(vaultId)`. Every Detect V2 method (`deidentifyString`, + * `deidentifyFile`, `reidentifyString`, `reidentifyFile`, `getRun`, + * `checkGuardrails`) is inherited from the shared `BaseDetectController` in + * `skyflow-common`; the flowvault `VaultClient` supplies the vault URL, vault + * id, credentials, log level and the per-vault timeout / retry overrides. + * + * This layer owns the vault-specific types: `deidentifyFile` takes the flowvault + * `DeidentifySourceType` (`BASE64` or `PRESIGNED_URL`; no skyflow id), and a run's or + * re-identification's `outputType` is resolved to the same enums, so a value the flowvault + * SDK does not offer comes back as `null`. A `VAULT_TOKEN` entity's `destination` is a token + * group name, and `RedactionLevel` rules use `tokenGroupName` and `redactionPattern`. + */ +class DetectController extends BaseDetectController { + constructor(client: VaultClient) { + super(client); + } + + protected toDeidentifySource(value: string | null | undefined): DeidentifySourceType | null { + return BaseDetectController.enumOrNull(DeidentifySourceType, value); + } + + protected toReidentifySource(value: string | null | undefined): ReidentifySourceType | null { + return BaseDetectController.enumOrNull(ReidentifySourceType, value); + } +} + +export default DetectController; diff --git a/flowvault/src/detect/types.ts b/flowvault/src/detect/types.ts new file mode 100644 index 00000000..1a93b91b --- /dev/null +++ b/flowvault/src/detect/types.ts @@ -0,0 +1,35 @@ +// Detect V2 types whose values depend on the vault type. These are the flowvault versions; the +// skyvault SDK ships its own. The shared request and response models in skyflow-common are typed +// against the contracts `DeidentifyFileSource`, `ReidentifyFileSource` and `RedactionRule`, which +// these satisfy. +import type { RedactionRule } from 'skyflow-common'; + +/** + * How the input file of `deidentifyFile` is supplied, and how a run delivers its outputs. + * Flowvault vaults accept a base64 string or a presigned URL; a skyflow id is not a source on + * flowvault. A local `file` or `filePath` is always sent as `BASE64`. + */ +export enum DeidentifySourceType { + BASE64 = 'BASE64', + PRESIGNED_URL = 'PRESIGNED_URL', +} + +/** How the input file of `reidentifyFile` is supplied, and how its outputs are delivered. */ +export enum ReidentifySourceType { + BASE64 = 'BASE64', + PRESIGNED_URL = 'PRESIGNED_URL', +} + +/** + * One redaction rule for `reidentifyString` and `reidentifyFile` on a flowvault vault: the + * token group the rule applies to and the name of a redaction pattern configured in the vault. + * Both fields are optional on the API; the server rejects combinations it does not support. + */ +export interface RedactionLevel extends RedactionRule { + /** Token group the rule applies to. */ + tokenGroupName?: string; + /** Name of a redaction pattern configured in the vault. */ + redactionPattern?: string; + entityName?: never; + redactionType?: never; +} diff --git a/flowvault/src/index.ts b/flowvault/src/index.ts index a6756510..380e219f 100644 --- a/flowvault/src/index.ts +++ b/flowvault/src/index.ts @@ -6,6 +6,11 @@ export * from 'skyflow-common'; // FlowVault facade. export { default as Skyflow, default } from './skyflow'; +// Detect V2 types whose values depend on the vault type (the shared request, response and +// configuration models come from skyflow-common above). +export { DeidentifySourceType, ReidentifySourceType } from './detect/types'; +export type { RedactionLevel } from './detect/types'; + // Request classes. export { default as InsertRequest } from './model/request/insert'; export { default as GetRequest } from './model/request/get'; diff --git a/flowvault/src/skyflow/index.ts b/flowvault/src/skyflow/index.ts index 244f7c8d..fbfa5762 100644 --- a/flowvault/src/skyflow/index.ts +++ b/flowvault/src/skyflow/index.ts @@ -11,6 +11,7 @@ import { } from 'skyflow-common'; import VaultClient from '../client'; import VaultController from '../controller/vault'; +import DetectController from '../controller/detect'; import { SDK_NAME, SDK_VERSION } from '../version'; // Stamp the SDK identity so telemetry and errors self-report FlowVault. @@ -35,13 +36,14 @@ function getFlowVaultURL(clusterId: string, env: Env): string { interface ClientEntry extends BaseVaultEntry { vaultController: VaultController; + detectController: DetectController; } /** * Skyflow is the FlowVault SDK facade. Vault-config / credential / log-level - * management comes from the shared `BaseSkyflow` in `common`; this layer only - * builds its own VaultClient + VaultController and exposes `vault(vaultId?)`. - * (FlowVault has no connections or detect.) + * management comes from the shared `BaseSkyflow` in `common`; this layer + * builds its own VaultClient plus the vault and detect controllers and exposes + * `vault(vaultId?)` and `detect(vaultId?)`. (FlowVault has no connections.) */ class Skyflow extends BaseSkyflow { constructor(config: SkyflowConfig) { @@ -62,12 +64,17 @@ class Skyflow extends BaseSkyflow { { timeout: config.timeout, maxRetries: config.maxRetries }, ); const vaultController = new VaultController(client); - return { config, client, vaultController }; + const detectController = new DetectController(client); + return { config, client, vaultController, detectController }; } vault(vaultId?: string): VaultController { return this.resolveVaultEntry(vaultId).vaultController; } + + detect(vaultId?: string): DetectController { + return this.resolveVaultEntry(vaultId).detectController; + } } export default Skyflow; diff --git a/flowvault/test/controller/detect.test.ts b/flowvault/test/controller/detect.test.ts new file mode 100644 index 00000000..e45fcf3c --- /dev/null +++ b/flowvault/test/controller/detect.test.ts @@ -0,0 +1,100 @@ +jest.mock('skyflow-common', () => { + const actual = jest.requireActual('skyflow-common'); + return { ...actual, getBearerToken: jest.fn() }; +}); + +import { + AuthType, + BaseDetectController, + DeidentifySourceType, + Env, + GetRunRequest, + LogLevel, + ReidentifyFileRequest, + ReidentifySourceType, + Skyflow, + getBearerToken, +} from '../../src'; +import DetectController from '../../src/controller/detect'; +import VaultClient from '../../src/client'; +import { silenceConsole } from '../helpers'; + +silenceConsole(); + +const getBearerTokenMock = getBearerToken as unknown as jest.Mock; + +describe('flowvault DetectController', () => { + it('extends the shared BaseDetectController and exposes every Detect V2 method', () => { + const client = new VaultClient('https://abc.skyvault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + const controller = new DetectController(client); + expect(controller).toBeInstanceOf(BaseDetectController); + for (const method of ['deidentifyString', 'deidentifyFile', 'reidentifyString', 'reidentifyFile', 'getRun', 'checkGuardrails']) { + expect(typeof (controller as unknown as Record)[method]).toBe('function'); + } + }); + + it('VaultClient.resolveAuthToken returns the token resolved by getToken', async () => { + getBearerTokenMock.mockResolvedValue({ key: 'tok-1', type: AuthType.TOKEN }); + const client = new VaultClient('https://abc.skyvault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + await expect(client.resolveAuthToken()).resolves.toBe('tok-1'); + expect(getBearerTokenMock).toHaveBeenCalledTimes(1); + }); + + it('Skyflow.detect(id) and detect() return the DetectController of the vault', () => { + const sky = new Skyflow({ + vaultConfigs: [ + { vaultId: 'vault123', clusterId: 'abc', env: Env.PROD, credentials: { apiKey: 'sky-key' } }, + { vaultId: 'vault456', clusterId: 'abc', env: Env.PROD, credentials: { apiKey: 'sky-key' } }, + ], + logLevel: LogLevel.ERROR, + }); + expect(sky.detect('vault123')).toBeInstanceOf(DetectController); + expect(sky.detect('vault456')).toBeInstanceOf(DetectController); + expect(sky.detect()).toBe(sky.detect('vault123')); + expect(sky.detect('vault123')).not.toBe(sky.detect('vault456')); + }); + + it('ships the flowvault source enums: no SKYFLOW_ID for de-identification, none for re-identification', () => { + expect(Object.values(DeidentifySourceType)).toEqual(['BASE64', 'PRESIGNED_URL']); + expect(Object.values(ReidentifySourceType)).toEqual(['BASE64', 'PRESIGNED_URL']); + expect((DeidentifySourceType as Record).SKYFLOW_ID).toBeUndefined(); + }); + + it('resolves run and re-identification output types onto the flowvault enums, unknown values to null', () => { + const client = new VaultClient('https://abc.skyvault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + const controller = new DetectController(client) as unknown as { + toDeidentifySource(value: string | null | undefined): DeidentifySourceType | null; + toReidentifySource(value: string | null | undefined): ReidentifySourceType | null; + }; + expect(controller.toDeidentifySource('BASE64')).toBe(DeidentifySourceType.BASE64); + expect(controller.toDeidentifySource('PRESIGNED_URL')).toBe(DeidentifySourceType.PRESIGNED_URL); + // A skyvault-only value is not something a flowvault user can be handed. + expect(controller.toDeidentifySource('SKYFLOW_ID')).toBeNull(); + expect(controller.toDeidentifySource(null)).toBeNull(); + expect(controller.toReidentifySource('PRESIGNED_URL')).toBe(ReidentifySourceType.PRESIGNED_URL); + expect(controller.toReidentifySource('SKYFLOW_ID')).toBeNull(); + expect(controller.toReidentifySource(undefined)).toBeNull(); + }); + + it('getRun and reidentifyFile surface the resolved output type', async () => { + const client = new VaultClient('https://abc.skyvault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + client.resolveAuthToken = jest.fn().mockResolvedValue('tok'); + const controller = new DetectController(client); + const files = { getRunV2: jest.fn(), reidentifyFileV2: jest.fn() }; + (controller as unknown as { files(): unknown }).files = () => files; + + files.getRunV2.mockResolvedValue({ status: 'SUCCESS', outputType: 'PRESIGNED_URL', output: [] }); + const run = await controller.getRun(new GetRunRequest('run-1')); + expect(run.outputType).toBe(DeidentifySourceType.PRESIGNED_URL); + + files.getRunV2.mockResolvedValue({ status: 'SUCCESS', outputType: 'SKYFLOW_ID', output: [] }); + expect((await controller.getRun(new GetRunRequest('run-2'))).outputType).toBeNull(); + + files.reidentifyFileV2.mockResolvedValue({ status: 'SUCCESS', outputType: 'BASE64', output: [] }); + const reid = await controller.reidentifyFile( + new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'YWJj', { redactionLevel: [{ tokenGroupName: 'NAME', redactionPattern: 'mask1' }] }), + ); + expect(reid.outputType).toBe(ReidentifySourceType.BASE64); + expect(files.reidentifyFileV2.mock.calls[0][0].redactionLevel).toEqual([{ tokenGroupName: 'NAME', redactionPattern: 'mask1' }]); + }); +}); diff --git a/samples/README.md b/samples/README.md index 90f49e8a..6c7adf36 100644 --- a/samples/README.md +++ b/samples/README.md @@ -51,6 +51,8 @@ Sample files are organized by API type: | `` | Actual values to insert into your vault | | ``, ``, etc. | Actual tokens from your vault | | `` | Your connection URL | + | `` | The id of a Detect configuration created through the management API | + | `` | A run id returned by a prior `deidentifyFile` call | 3. Run the sample: @@ -80,12 +82,13 @@ Sample files are organized by API type: - [`data-residency.ts`](vault-api/data-residency.ts) - Multi-region vault configuration **Detect API samples** ([`detect-api/`](detect-api/)): -- [`deidentify-text.ts`](detect-api/deidentify-text.ts) - Anonymize text data -- [`deidentify-file.ts`](detect-api/deidentify-file.ts) - Anonymize a file using a File object -- [`deidentify-file-with-filepath.ts`](detect-api/deidentify-file-with-filepath.ts) - Anonymize a file using a file path -- [`deidentify-file-with-filepath-async.ts`](detect-api/deidentify-file-with-filepath-async.ts) - Anonymize a file asynchronously (poll for result) -- [`reidentify-text.ts`](detect-api/reidentify-text.ts) - Restore original values from de-identified text -- [`get-detect-run.ts`](detect-api/get-detect-run.ts) - Check the status of an async de-identify operation +- [`deidentify-string.ts`](detect-api/deidentify-string.ts) - De-identify text with a stored or inline Detect configuration +- [`deidentify-file.ts`](detect-api/deidentify-file.ts) - Submit a file (local path, skyflow id or presigned URL) and get a run id +- [`deidentify-file-async.ts`](detect-api/deidentify-file-async.ts) - Submit a file and let the SDK poll the run, writing outputs to disk +- [`reidentify-string.ts`](detect-api/reidentify-string.ts) - Restore original values from de-identified text, with redaction rules +- [`reidentify-file.ts`](detect-api/reidentify-file.ts) - Restore original values in a tokenized file +- [`get-run.ts`](detect-api/get-run.ts) - Check the status and outputs of a file de-identification run +- [`check-guardrails.ts`](detect-api/check-guardrails.ts) - Screen text for toxicity and denied topics **Service Account samples** ([`service-account/`](service-account/)): - [`token-generation-example.ts`](service-account/token-generation-example.ts) - Generate bearer tokens diff --git a/samples/detect-api/check-guardrails.ts b/samples/detect-api/check-guardrails.ts new file mode 100644 index 00000000..4910edf6 --- /dev/null +++ b/samples/detect-api/check-guardrails.ts @@ -0,0 +1,82 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + CheckGuardrailsRequest, + CheckGuardrailsResponse, + GuardrailsValidation, +} from 'skyflow-node'; + +/** + * Skyflow Detect: check guardrails + * + * checkGuardrails screens text for toxicity and for a list of denied topics, which makes it a + * natural pre-LLM check. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Screen a prompt for toxicity and denied topics + * 3. Block the prompt when the validation fails + * 4. Handle errors + */ +async function performCheckGuardrails() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Screen the text. checkToxicity and denyTopics are optional and sent only when set. + const userPrompt = 'Tell me which candidate to vote for.'; + const request = new CheckGuardrailsRequest(userPrompt, { + checkToxicity: true, + denyTopics: ['politics', 'medical advice'], + }); + const response: CheckGuardrailsResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .checkGuardrails(request); + + // Step 5: Act on the result. toxic and deniedTopic are null when that check did not run. + console.log('Validation:', response.validation); + console.log('Toxic:', response.toxic, 'Denied topic:', response.deniedTopic); + if (response.validation === GuardrailsValidation.FAILED) { + console.log('Prompt blocked before reaching the model.'); + } else { + console.log('Prompt allowed.'); + } + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the check guardrails function +performCheckGuardrails(); diff --git a/samples/detect-api/deidentify-file-async.ts b/samples/detect-api/deidentify-file-async.ts new file mode 100644 index 00000000..6e107c0d --- /dev/null +++ b/samples/detect-api/deidentify-file-async.ts @@ -0,0 +1,126 @@ +import * as fs from 'fs'; +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + DeidentifyFileRequest, + DeidentifyFileResponse, + DetectRunStatus, + DetectConfiguration, + EntityType, + DeidentificationType, + MaskingMethod, + PdfProcessingMode, +} from 'skyflow-node'; + +/** + * Skyflow Detect: de-identify a file and poll the run to completion + * + * This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Submit a local file with pollOptions so the SDK polls the run for you + * 3. Write the processed outputs to an output directory on success + * 4. Submit an in-memory File object (Node.js 20+) with an inline configuration + * 5. Handle a run that is still in progress when the polling budget runs out + */ +async function performDeidentifyFileAndPoll() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Submit and poll. The SDK polls with exponential backoff (1, 2, 4, 8, 16, then + // 32 seconds between lookups) until the run is SUCCESS, FAILED or UNKNOWN, or until + // waitTime (default 60 s, max 300) or maxAttempts (default 23) is used up. + const request = new DeidentifyFileRequest({ + filePath: '', // e.g. /data/intake/patient-notes.pdf + configurationId: '', + pollOptions: { waitTime: 120, maxAttempts: 10 }, + outputDirectory: '', // optional; created if missing, default: working directory + }); + const response: DeidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(request); + + handleRun(response); + + // Step 5 (alternative): an in-memory File with an inline configuration + const buffer = fs.readFileSync(''); + const file = new File([new Uint8Array(buffer)], 'patient-notes.pdf'); + const configuration: DetectConfiguration = { + detect: { + entities: [ + { entityType: EntityType.NAME, deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER }, + { entityType: EntityType.ALL }, + ], + }, + media: { + image: { maskingMethod: MaskingMethod.BLUR, outputOcrText: true }, + document: { pdf: { processingMode: PdfProcessingMode.OCR } }, + }, + }; + const fileObjectResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(new DeidentifyFileRequest({ file, configuration, pollOptions: {} })); + + handleRun(fileObjectResponse); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +function handleRun(response: DeidentifyFileResponse) { + switch (response.status) { + case DetectRunStatus.SUCCESS: + // For local-file requests the outputs are already on disk as + // processed-. and processed-entities-.json. + console.log('Run succeeded:', response.runId); + for (const output of response.output) { + console.log(`${output.processedFileType} (${output.processedFileExtension})`); + } + console.log('Metrics:', response.metrics); + break; + case DetectRunStatus.FAILED: + console.error('Run failed:', response.message); + break; + default: + // IN_PROGRESS or QUEUED: the polling budget ran out. Resume with getRun(runId). + console.log(`Run ${response.runId} is still ${response.status}; poll it with getRun.`); + break; + } +} + +// Invoke the de-identify and poll function +performDeidentifyFileAndPoll(); diff --git a/samples/detect-api/deidentify-file-with-filepath-async.ts b/samples/detect-api/deidentify-file-with-filepath-async.ts deleted file mode 100644 index 2841b57e..00000000 --- a/samples/detect-api/deidentify-file-with-filepath-async.ts +++ /dev/null @@ -1,145 +0,0 @@ -import { - Credentials, - Env, - LogLevel, - Skyflow, - SkyflowConfig, - SkyflowError, - DeidentifyFileRequest, - DeidentifyFileOptions, - DetectEntities, - MaskingMethod, - DetectOutputTranscription, - TokenFormat, - TokenType, - Transformations, - Bleep, - VaultConfig, - DeidentifyFileResponse, - FileInput, - } from 'skyflow-node'; - - /** - * Skyflow Deidentify File Example - * - * This sample demonstrates how to use all available options for deidentifying files. - * Supported file types: images (jpg, png, etc.), pdf, audio (mp3, wav), documents, spreadsheets, presentations, structured text. - * - * Note: File deidentification requires Node.js version 20 or above. - */ - - async function performDeidentifyFileAsync() { - // Step 1: Configure Credentials - const credentials: Credentials = { - path: 'path-to-credentials-json', // Replace with your actual token or path to credentials file - }; - - // Step 2: Configure Vault - const primaryVaultConfig: VaultConfig = { - vaultId: '', - clusterId: '', - env: Env.PROD, - credentials: credentials, - }; - - // Step 3: Configure Skyflow Client - const skyflowConfig: SkyflowConfig = { - vaultConfigs: [primaryVaultConfig], - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. - }; - - // Initialize Skyflow Client - const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - - // Step 4: Prepare Deidentify File Request - // Replace with your file object (e.g., from fs.readFileSync or browser File API) - - const filePath: string = ''; // Replace with the path to your file - - // Pass either file object or file path, but not both. - const fileInput: FileInput = { - filePath - }; - - const deidentifyFile = new DeidentifyFileRequest(fileInput); - - // Step 5: Configure DeidentifyFileOptions - const options = new DeidentifyFileOptions(); - - // Entities to detect and deidentify - options.setEntities([DetectEntities.SSN, DetectEntities.CREDIT_CARD]); - - // Allowlist regex patterns (entities matching these will NOT be deidentified) - options.setAllowRegexList(['']); - - // Restrict deidentification to entities matching these regex patterns - options.setRestrictRegexList(['']); - - // Token format for deidentified entities - const tokenFormat = new TokenFormat(); - tokenFormat.setDefault(TokenType.ENTITY_ONLY); - options.setTokenFormat(tokenFormat); - - // Custom transformations for entities - // const transformations = new Transformations(); // Transformations cannot be applied to Documents, Images, or PDFs file formats. - // transformations.setShiftDays({ - // max: 30, - // min: 10, - // entities: [DetectEntities.SSN], - // }); - // options.setTransformations(transformations); - - // Output directory for saving the deidentified file - // Providing an output directory is not supported in Cloudflare Workers - options.setOutputDirectory(''); // Replace with your output directory - - // Wait time for response (max 64 seconds) - options.setWaitTime(15); - - // --- Image Options (apply when file is an image) --- - // options.setOutputProcessedImage(true); // Include processed image in output - // options.setOutputOcrText(true); // Include OCR text in response - // options.setMaskingMethod(MaskingMethod.Blackbox); // Masking method for image entities - - // --- PDF Options (apply when file is a PDF) --- - // options.setPixelDensity(1.5); // Pixel density for PDF processing - // options.setMaxResolution(2000); // Max resolution for PDF - - // --- Audio Options (apply when file is audio) --- - // options.setOutputProcessedAudio(true); // Include processed audio in output - // options.setOutputTranscription(DetectOutputTranscription.PLAINTEXT_TRANSCRIPTION); // Type of transcription - - // Bleep audio configuration - // const bleep = new Bleep(); - // bleep.setGain(5); // Loudness in dB - // bleep.setFrequency(1000); // Pitch in Hz - // bleep.setStartPadding(0.1); // Padding at start in seconds - // bleep.setStopPadding(0.2); // Padding at end in seconds - // options.setBleep(bleep); - - // Step 6: Call deidentifyFile API - skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyFile(deidentifyFile, options) - .then((response: DeidentifyFileResponse) => { - // Handle Successful Response - console.log('Deidentify File Response:', response); - console.log('Deidentified File:', response.file); - console.log('Deidentified File base64:', response.fileBase64); - }) - .catch((error) => { - // Comprehensive Error Handling - if (error instanceof SkyflowError) { - console.error('Skyflow Specific Error:', { - code: error.error?.httpCode, - message: error.message, - details: error.error?.details, - }); - } else { - console.error('Unexpected Error:', JSON.stringify(error)); - } - }); -} - -// Invoke the deidentify file function -performDeidentifyFileAsync(); diff --git a/samples/detect-api/deidentify-file-with-filepath.ts b/samples/detect-api/deidentify-file-with-filepath.ts deleted file mode 100644 index b0638776..00000000 --- a/samples/detect-api/deidentify-file-with-filepath.ts +++ /dev/null @@ -1,150 +0,0 @@ -import { - Credentials, - Env, - LogLevel, - Skyflow, - SkyflowConfig, - SkyflowError, - DeidentifyFileRequest, - DeidentifyFileOptions, - DetectEntities, - MaskingMethod, - DetectOutputTranscription, - TokenFormat, - TokenType, - Transformations, - Bleep, - VaultConfig, - DeidentifyFileResponse, - FileInput, - } from 'skyflow-node'; - import fs from 'fs'; - - /** - * Skyflow Deidentify File Example - * - * This sample demonstrates how to use all available options for de-identifying files. - * Supported file types: images (jpg, png, etc.), pdf, audio (mp3, wav), documents, spreadsheets, presentations, structured text. - * - * Note: File de-identification requires Node.js version 20 or above. - */ - - async function performDeidentifyFile() { - try { - // Step 1: Configure Credentials - const credentials: Credentials = { - path: 'path-to-credentials-json', // Path to credentials file - }; - - // Step 2: Configure Vault - const primaryVaultConfig: VaultConfig = { - vaultId: '', - clusterId: '', - env: Env.PROD, - credentials: credentials, - }; - - // Step 3: Configure Skyflow Client - const skyflowConfig: SkyflowConfig = { - vaultConfigs: [primaryVaultConfig], - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. - }; - - // Initialize Skyflow Client - const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - - // Step 4: Prepare Deidentify File Request - // Replace with your file object (e.g., from fs.readFileSync or browser File API) - - const filePath: string = ''; // Replace with the path to your file - - // Pass wither file object or file path, but not both. - const fileInput: FileInput = { - filePath - } - - const deidentifyFile = new DeidentifyFileRequest(fileInput); - - // Step 5: Configure DeidentifyFileOptions - const options = new DeidentifyFileOptions(); - - // Entities to detect and deidentify - options.setEntities([DetectEntities.SSN, DetectEntities.CREDIT_CARD]); - - // Allowlist regex patterns (entities matching these will NOT be deidentified) - options.setAllowRegexList(['']); - - // Restrict de-identification to entities matching these regex patterns - options.setRestrictRegexList(['']); - - // Token format for deidentified entities - const tokenFormat = new TokenFormat(); - tokenFormat.setDefault(TokenType.ENTITY_ONLY); - options.setTokenFormat(tokenFormat); - - // Custom transformations for entities - // const transformations = new Transformations(); // Transformations cannot be applied to Documents, Images, or PDFs file formats. - // transformations.setShiftDays({ - // max: 30, - // min: 10, - // entities: [DetectEntities.SSN], - // }); - // options.setTransformations(transformations); - - // Output directory for saving the deidentified file - // Providing an output directory is not supported in Cloudflare Workers - options.setOutputDirectory(''); // Replace with your output directory - - // Wait time for response (max 64 seconds) - options.setWaitTime(15); - - // --- Image Options (apply when file is an image) --- - // options.setOutputProcessedImage(true); // Include processed image in output - // options.setOutputOcrText(true); // Include OCR text in response - // options.setMaskingMethod(MaskingMethod.Blackbox); // Masking method for image entities - - // --- PDF Options (apply when file is a PDF) --- - // options.setPixelDensity(1.5); // Pixel density for PDF processing - // options.setMaxResolution(2000); // Max resolution for PDF - - // --- Audio Options (apply when file is audio) --- - // options.setOutputProcessedAudio(true); // Include processed audio in output - // options.setOutputTranscription(DetectOutputTranscription.PLAINTEXT_TRANSCRIPTION); // Type of transcription - - // Bleep audio configuration - // const bleep = new Bleep(); - // bleep.setGain(5); // Loudness in dB - // bleep.setFrequency(1000); // Pitch in Hz - // bleep.setStartPadding(0.1); // Padding at start in seconds - // bleep.setStopPadding(0.2); // Padding at end in seconds - // options.setBleep(bleep); - - - // Step 6: Call deidentifyFile API - const response: DeidentifyFileResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyFile(deidentifyFile, options); - - // Handle Successful Response - console.log('Deidentify File Response:', response); - console.log('Deidentified File:', response.file); - console.log('Deidentified File base64:', response.fileBase64); - - } catch (error) { - // Comprehensive Error Handling - if (error instanceof SkyflowError) { - console.error('Skyflow Specific Error:', { - httpCode: error.error?.httpCode, - grpcCode: error.error?.grpcCode, - httpStatus: error.error?.httpStatus, - message: error.message, - details: error.error?.details, - }); - } else { - console.error('Unexpected Error:', JSON.stringify(error)); - } - } - } - - // Invoke the deidentify file function - performDeidentifyFile(); \ No newline at end of file diff --git a/samples/detect-api/deidentify-file.ts b/samples/detect-api/deidentify-file.ts index ace759bf..2e36bc77 100644 --- a/samples/detect-api/deidentify-file.ts +++ b/samples/detect-api/deidentify-file.ts @@ -1,150 +1,91 @@ import { - Credentials, - Env, - LogLevel, - Skyflow, - SkyflowConfig, - SkyflowError, - DeidentifyFileRequest, - DeidentifyFileOptions, - DetectEntities, - MaskingMethod, - DetectOutputTranscription, - TokenFormat, - TokenType, - Transformations, - Bleep, - VaultConfig, - DeidentifyFileResponse, - FileInput, -} from 'skyflow-node'; -import fs from 'fs'; + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + DeidentifyFileRequest, + DeidentifyFileResponse, + DeidentifySourceType, + FileDataFormat, +} from 'skyflow-node'; /** - * Skyflow Deidentify File Example + * Skyflow Detect: de-identify a file (submit only) * - * This sample demonstrates how to use all available options for de-identifying files. - * Supported file types: images (jpg, png, etc.), pdf, audio (mp3, wav), documents, spreadsheets, presentations, structured text. + * File de-identification is asynchronous on the API. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Submit a local file for de-identification and receive a run id + * 3. Submit a file by skyflow id or presigned URL instead of a local file + * 4. Handle errors * - * Note: File de-identification requires Node.js version 20 or above. + * Retrieve the result later with getRun (see get-run.ts), or let the SDK poll for you + * (see deidentify-file-async.ts). */ - async function performDeidentifyFile() { - try { - // Step 1: Configure Credentials - const credentials: Credentials = { - path: 'path-to-credentials-json', // Path to credentials file - }; - - // Step 2: Configure Vault - const primaryVaultConfig: VaultConfig = { - vaultId: '', - clusterId: '', - env: Env.PROD, - credentials: credentials, - }; - - // Step 3: Configure Skyflow Client - const skyflowConfig: SkyflowConfig = { - vaultConfigs: [primaryVaultConfig], - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. - }; - - // Initialize Skyflow Client - const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - - // Step 4: Prepare Deidentify File Request - // Replace with your file object (e.g., from fs.readFileSync or browser File API) - const filePath = ''; // Replace with the path to your file - const buffer = fs.readFileSync(filePath); - const file = new File([buffer], filePath); - - // Pass wither file object or file path, but not both. - const fileInput: FileInput = { - file + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Submit a local file. It is read, base64-encoded and sent as BASE64; + // dataFormat is inferred from the extension unless you set it. + const localFileRequest = new DeidentifyFileRequest({ + filePath: '', // e.g. /data/intake/patient-notes.pdf + configurationId: '', // or configuration: { ... } for an inline configuration + }); + const submitted: DeidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(localFileRequest); + + console.log('Run id:', submitted.runId); // status, output and metrics are null until polled + // Pass submitted.runId to getRun (see get-run.ts) to retrieve the result. + + // Step 5 (alternative): submit a file the vault already holds, or one at a presigned URL + const remoteFileRequest = new DeidentifyFileRequest({ + dataSource: DeidentifySourceType.SKYFLOW_ID, // or DeidentifySourceType.PRESIGNED_URL + value: '', // or the presigned URL + dataFormat: FileDataFormat.PDF, + configurationId: '', + }); + const remoteSubmitted = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyFile(remoteFileRequest); + console.log('Run id (remote file):', remoteSubmitted.runId); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } } - - const deidentifyFile = new DeidentifyFileRequest(fileInput); - - // Step 5: Configure DeidentifyFileOptions - const options = new DeidentifyFileOptions(); - - // Entities to detect and deidentify - options.setEntities([DetectEntities.SSN, DetectEntities.CREDIT_CARD]); - - // Allowlist regex patterns (entities matching these will NOT be deidentified) - options.setAllowRegexList(['']); - - // Restrict de-identification to entities matching these regex patterns - options.setRestrictRegexList(['']); - - // Token format for deidentified entities - const tokenFormat = new TokenFormat(); - tokenFormat.setDefault(TokenType.ENTITY_ONLY); - options.setTokenFormat(tokenFormat); - - // Custom transformations for entities - // const transformations = new Transformations(); // Transformations cannot be applied to Documents, Images, or PDFs file formats. - // transformations.setShiftDays({ - // max: 30, - // min: 10, - // entities: [DetectEntities.SSN], - // }); - // options.setTransformations(transformations); - - // Output directory for saving the deidentified file - options.setOutputDirectory(''); // Replace with your output directory - - // Wait time for response (max 64 seconds) - options.setWaitTime(15); - - // --- Image Options (apply when file is an image) --- - // options.setOutputProcessedImage(true); // Include processed image in output - // options.setOutputOcrText(true); // Include OCR text in response - // options.setMaskingMethod(MaskingMethod.Blackbox); // Masking method for image entities - - // --- PDF Options (apply when file is a PDF) --- - // options.setPixelDensity(1.5); // Pixel density for PDF processing - // options.setMaxResolution(2000); // Max resolution for PDF - - // --- Audio Options (apply when file is audio) --- - // options.setOutputProcessedAudio(true); // Include processed audio in output - // options.setOutputTranscription(DetectOutputTranscription.PLAINTEXT_TRANSCRIPTION); // Type of transcription - - // Bleep audio configuration - // const bleep = new Bleep(); - // bleep.setGain(5); // Loudness in dB - // bleep.setFrequency(1000); // Pitch in Hz - // bleep.setStartPadding(0.1); // Padding at start in seconds - // bleep.setStopPadding(0.2); // Padding at end in seconds - // options.setBleep(bleep); - - - // Step 6: Call deidentifyFile API - const response: DeidentifyFileResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyFile(deidentifyFile, options); - - // Handle Successful Response - console.log('Deidentify File Response:', response); - console.log('Deidentified File:', response.file); - console.log('Deidentified File base64:', response.fileBase64); - - } catch (error) { - // Comprehensive Error Handling - if (error instanceof SkyflowError) { - console.error('Skyflow Specific Error:', { - httpCode: error.error?.httpCode, - grpcCode: error.error?.grpcCode, - httpStatus: error.error?.httpStatus, - message: error.message, - details: error.error?.details, - }); - } else { - console.error('Unexpected Error:', JSON.stringify(error)); - } - } } -// Invoke the deidentify file function -performDeidentifyFile(); \ No newline at end of file +// Invoke the de-identify file function +performDeidentifyFile(); diff --git a/samples/detect-api/deidentify-string.ts b/samples/detect-api/deidentify-string.ts new file mode 100644 index 00000000..2ae83e3d --- /dev/null +++ b/samples/detect-api/deidentify-string.ts @@ -0,0 +1,109 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + DeidentifyStringRequest, + DeidentifyStringResponse, + DetectConfiguration, + EntityType, + DeidentificationType, + ReturnEntitiesType, +} from 'skyflow-node'; + +/** + * Skyflow Detect: de-identify a string + * + * This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. De-identify text with a stored Detect configuration + * 3. De-identify text with an inline Detect configuration + * 4. Read the processed text, detected entities and metrics + * 5. Handle errors + */ +async function performDeidentifyString() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + const text = 'My name is John Doe, and my email is johndoe@acme.com.'; + + // Step 4: De-identify with a stored configuration (created through the management API) + const storedRequest = new DeidentifyStringRequest(text, { + configurationId: '', + }); + const storedResponse: DeidentifyStringResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyString(storedRequest); + console.log('Processed text (stored configuration):', storedResponse.processedText); + // My name is [NAME_1], and my email is [EMAIL_ADDRESS_1]. + + // Step 5: De-identify with an inline configuration + const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '
.', // table.column that stores the value; must exist + }, + { + entityType: EntityType.ALL, + deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, + }, + ], + skip: ['Skyflow'], // never treat these matches as sensitive + returnEntities: ReturnEntitiesType.EXCLUDE_SENSITIVE_DATA, + }, + }; + const inlineRequest = new DeidentifyStringRequest(text, { configuration }); + const inlineResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyString(inlineRequest); + + console.log('Processed text (inline configuration):', inlineResponse.processedText); + for (const entity of inlineResponse.entities) { + console.log( + `${entity.entityType} -> ${entity.token} [${entity.location?.startIndex}, ${entity.location?.endIndex}]`, + ); + } + console.log('Word count:', inlineResponse.metrics?.wordCount); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the de-identify string function +performDeidentifyString(); diff --git a/samples/detect-api/deidentify-text.ts b/samples/detect-api/deidentify-text.ts deleted file mode 100644 index dcc09737..00000000 --- a/samples/detect-api/deidentify-text.ts +++ /dev/null @@ -1,122 +0,0 @@ -import { - Credentials, - Env, - LogLevel, - Skyflow, - SkyflowConfig, - VaultConfig, - SkyflowError, - DeidentifyTextRequest, - DeidentifyTextOptions, - TokenFormat, - TokenType, - Transformations, - DetectEntities, - DeidentifyTextResponse -} from 'skyflow-node'; - -/** - * Skyflow Deidentify Text Example - * - * This example demonstrates how to: - * 1. Configure credentials - * 2. Set up vault configuration - * 3. Create a deidentify text request - * 4. Use all available options for de-identification - * 5. Handle response and errors - */ - -async function performDeidentifyText() { - try { - // Step 1: Configure Credentials - const credentials: Credentials = { - path: 'path-to-credentials-json', // Path to credentials file - }; - - // Step 2: Configure Vault - const primaryVaultConfig: VaultConfig = { - vaultId: '', // Unique vault identifier - clusterId: '', // From vault URL - env: Env.PROD, // Deployment environment - credentials: credentials // Authentication method - }; - - // Step 3: Configure Skyflow Client - const skyflowConfig: SkyflowConfig = { - vaultConfigs: [primaryVaultConfig], - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. - }; - - // Initialize Skyflow Client - const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - - // Step 4: Prepare Deidentify Text Request - const deidentifyTextRequest = new DeidentifyTextRequest( - 'My SSN is 123-45-6789 and my card is 4111 1111 1111 1111.', // Text to be deidentified - ); - - // Step 5: Configure DeidentifyTextOptions - const options = new DeidentifyTextOptions(); - - // setEntities: Specify which entities to deidentify - options.setEntities([DetectEntities.CREDIT_CARD, DetectEntities.SSN]); - - // Allowlist: regex patterns whose matches will NOT be de-identified - // options.setAllowRegexList(['']); - - // Denylist: restrict de-identification to only entities matching these patterns - // options.setRestrictRegexList(['']); - - // setTokenFormat: choose how de-identified entities are represented as tokens - const tokenFormat = new TokenFormat(); - - // Apply one token type as the default for all entity types - tokenFormat.setDefault(TokenType.VAULT_TOKEN); - - // --- Per-entity-type token format overrides (optional) --- - // Use vault tokens only for specific entity types - // tokenFormat.setVaultToken([DetectEntities.SSN, DetectEntities.CREDIT_CARD]); - - // Use an entity-unique counter token for specific entity types - // tokenFormat.setEntityUniqueCounter([DetectEntities.NAME]); - - // Use entity-only (no token) for specific entity types - // tokenFormat.setEntityOnly([DetectEntities.DATE_OF_BIRTH]); - - options.setTokenFormat(tokenFormat); - - // setTransformations: Specify custom transformations for entities - const transformations = new Transformations(); - transformations.setShiftDays({ - max: 30, // Maximum shift days - min: 30, // Minimum shift days - entities: [DetectEntities.DOB], // Entities to apply the shift - }); - options.setTransformations(transformations); - - // Step 6: Call deidentifyText API - const response: DeidentifyTextResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyText(deidentifyTextRequest, options); - - // Handle Successful Response - console.log('Deidentify Text Response:', response); - - } catch (error) { - // Comprehensive Error Handling - if (error instanceof SkyflowError) { - console.error('Skyflow Specific Error:', { - httpCode: error.error?.httpCode, - grpcCode: error.error?.grpcCode, - httpStatus: error.error?.httpStatus, - message: error.message, - details: error.error?.details, - }); - } else { - console.error('Unexpected Error:', JSON.stringify(error)); - } - } -} - -// Invoke the deidentify text function -performDeidentifyText(); \ No newline at end of file diff --git a/samples/detect-api/get-detect-run.ts b/samples/detect-api/get-detect-run.ts deleted file mode 100644 index ba4d9538..00000000 --- a/samples/detect-api/get-detect-run.ts +++ /dev/null @@ -1,78 +0,0 @@ -import { - Credentials, - Env, - LogLevel, - Skyflow, - SkyflowConfig, - VaultConfig, - SkyflowError, - GetDetectRunRequest, - DeidentifyFileResponse -} from 'skyflow-node'; - -/** - * Skyflow Get Detect Run Example - * - * This example demonstrates how to: - * 1. Configure credentials - * 2. Set up vault configuration - * 3. Create a get detect run request - * 4. Call getDetectRun to poll for file processing results - * 5. Handle response and errors - */ - -async function performGetDetectRun() { - try { - // Step 1: Configure Credentials - const credentials: Credentials = { - path: 'path-to-credentials-json', // Path to credentials file - }; - - // Step 2: Configure Vault - const primaryVaultConfig: VaultConfig = { - vaultId: '', // Unique vault identifier - clusterId: '', // From vault URL - env: Env.PROD, // Deployment environment - credentials: credentials // Authentication method - }; - - // Step 3: Configure Skyflow Client - const skyflowConfig: SkyflowConfig = { - vaultConfigs: [primaryVaultConfig], - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. - }; - - // Initialize Skyflow Client - const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - - // Step 4: Prepare GetDetectRunRequest - const getDetectRunRequest = new GetDetectRunRequest( - '', // Replace with the runId from deidentifyFile call - ); - - // Step 5: Call getDetectRun API - const response: DeidentifyFileResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .getDetectRun(getDetectRunRequest); - - // Handle Successful Response - console.log('Get Detect Run Response:', response); - - } catch (error) { - // Comprehensive Error Handling - if (error instanceof SkyflowError) { - console.error('Skyflow Specific Error:', { - httpCode: error.error?.httpCode, - grpcCode: error.error?.grpcCode, - httpStatus: error.error?.httpStatus, - message: error.message, - details: error.error?.details, - }); - } else { - console.error('Unexpected Error:', JSON.stringify(error)); - } - } -} - -// Invoke the get detect run function -performGetDetectRun(); \ No newline at end of file diff --git a/samples/detect-api/get-run.ts b/samples/detect-api/get-run.ts new file mode 100644 index 00000000..8c443d44 --- /dev/null +++ b/samples/detect-api/get-run.ts @@ -0,0 +1,90 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + GetRunRequest, + GetRunResponse, + DetectRunStatus, +} from 'skyflow-node'; + +/** + * Skyflow Detect: get a file run + * + * File de-identification is asynchronous: deidentifyFile returns a run id, and getRun retrieves + * the run's status and, once it has succeeded, its outputs. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Look up a run by id + * 3. Act on the run status and read its outputs + * 4. Handle errors + */ +async function performGetRun() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Look up the run returned by a prior deidentifyFile call + const request = new GetRunRequest(''); + const run: GetRunResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .getRun(request); + + // Step 5: Act on the status + switch (run.status) { + case DetectRunStatus.SUCCESS: + for (const output of run.output) { + // output.processedFile is base64 or a presigned URL, per run.outputType + // output.processedFileType is e.g. REDACTED_FILE or ENTITIES + // output.processedFileExtension is a FileDataFormat such as PDF + console.log(`${output.processedFileType} (${output.processedFileExtension})`); + } + console.log('Output type:', run.outputType); + console.log('Metrics:', run.metrics); + break; + case DetectRunStatus.FAILED: + console.error('Run failed:', run.message); + break; + default: + // QUEUED, IN_PROGRESS, UNKNOWN: poll again later + console.log(`Run ${run.runId} is ${run.status}; try again later.`); + break; + } + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the get run function +performGetRun(); diff --git a/samples/detect-api/reidentify-file.ts b/samples/detect-api/reidentify-file.ts new file mode 100644 index 00000000..cc34c485 --- /dev/null +++ b/samples/detect-api/reidentify-file.ts @@ -0,0 +1,95 @@ +import * as fs from 'fs'; +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + ReidentifyFileRequest, + ReidentifyFileResponse, + ReidentifySourceType, + FileDataFormat, + DetectRunStatus, + EntityType, + RedactionType, +} from 'skyflow-node'; + +/** + * Skyflow Detect: re-identify a file + * + * reidentifyFile is synchronous: the re-identified file comes back in the response. This + * example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Send a tokenized local file as base64 with optional redaction rules + * 3. Decode the re-identified output + * 4. Handle errors + */ +async function performReidentifyFile() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: Read the tokenized file and send it as BASE64. dataFormat is limited to the + // formats that support VAULT_TOKEN: TXT, CSV, JSON, JSONL, XML, DOCX, XLSX and PPTX. + const content = fs.readFileSync('').toString('base64'); + const request = new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { + dataFormat: FileDataFormat.TXT, + redactionLevel: [ + { entityName: EntityType.SSN, redactionType: RedactionType.MASKED }, + ], + }); + const response: ReidentifyFileResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyFile(request); + + // Step 5: Decode the output. processedFile is base64 or a presigned URL, per outputType. + if (response.status === DetectRunStatus.SUCCESS) { + for (const output of response.output) { + if (output.processedFile) { + const bytes = Buffer.from(output.processedFile, 'base64'); + fs.writeFileSync(`reidentified.${output.processedFileExtension?.toLowerCase()}`, bytes); + console.log(`Wrote ${output.processedFileType} (${bytes.length} bytes)`); + } + } + console.log('Metrics:', response.metrics); + } else { + console.error('Re-identification did not succeed:', response.status); + } + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the re-identify file function +performReidentifyFile(); diff --git a/samples/detect-api/reidentify-string.ts b/samples/detect-api/reidentify-string.ts new file mode 100644 index 00000000..1b7e3cbd --- /dev/null +++ b/samples/detect-api/reidentify-string.ts @@ -0,0 +1,86 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + ReidentifyStringRequest, + ReidentifyStringResponse, + EntityType, + RedactionType, +} from 'skyflow-node'; + +/** + * Skyflow Detect: re-identify a string + * + * Re-identification looks tokens up in the vault, so it applies to values that were + * de-identified with VAULT_TOKEN. This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. Re-identify tokenized text, restoring the original values + * 3. Apply redaction rules so some entities come back masked or redacted + * 4. Handle errors + */ +async function performReidentifyString() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + const tokenizedText = 'My name is [NAME_1], and my SSN is [SSN_1].'; + + // Step 4: Without rules every token is replaced by its original value + const plainRequest = new ReidentifyStringRequest(tokenizedText); + const plainResponse: ReidentifyStringResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyString(plainRequest); + console.log('Re-identified text:', plainResponse.processedText); + + // Step 5: Redaction rules. Each rule names an entity and how to render it. + const redactedRequest = new ReidentifyStringRequest(tokenizedText, { + redactionLevel: [ + { entityName: EntityType.NAME, redactionType: RedactionType.MASKED }, + { entityName: EntityType.SSN, redactionType: RedactionType.REDACTED }, + ], + }); + const redactedResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyString(redactedRequest); + console.log('Re-identified text with redaction rules:', redactedResponse.processedText); + console.log('Character count:', redactedResponse.metrics?.characterCount); + } catch (error) { + // Comprehensive error handling + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the re-identify string function +performReidentifyString(); diff --git a/samples/detect-api/reidentify-text.ts b/samples/detect-api/reidentify-text.ts deleted file mode 100644 index 8ecdcd44..00000000 --- a/samples/detect-api/reidentify-text.ts +++ /dev/null @@ -1,88 +0,0 @@ -import { - Credentials, - Env, - LogLevel, - Skyflow, - SkyflowConfig, - VaultConfig, - ReidentifyTextRequest, - ReidentifyTextOptions, - DetectEntities, - SkyflowError, - ReidentifyTextResponse -} from 'skyflow-node'; - -/** - * Skyflow Reidentify Text Example - * - * This example demonstrates how to: - * 1. Configure credentials - * 2. Set up vault configuration - * 3. Create a reidentify text request - * 4. Use all available options for re-identification - * 5. Handle response and errors - */ - -async function performReidentifyText() { - try { - // Step 1: Configure Credentials - const credentials: Credentials = { - path: 'path-to-credentials-json', // Path to credentials file - }; - - // Step 2: Configure Vault - const primaryVaultConfig: VaultConfig = { - vaultId: '', - clusterId: '', - env: Env.PROD, - credentials: credentials - }; - - // Step 3: Configure Skyflow Client - const skyflowConfig: SkyflowConfig = { - vaultConfigs: [primaryVaultConfig], - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. - }; - - // Initialize Skyflow Client - const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - - // Step 4: Prepare Reidentify Text Request - const reidentifyRequest = new ReidentifyTextRequest( - '' // The redacted text to reidentify - ); - - // Step 5: Configure ReidentifyTextOptions - const options = new ReidentifyTextOptions(); - - // Specify which entities to reidentify as redacted, masked, or plain text - options.setRedactedEntities([DetectEntities.NAME, DetectEntities.SSN]); - options.setMaskedEntities([DetectEntities.DOB]); - options.setPlainTextEntities([DetectEntities.PHONE_NUMBER]); - - // Step 6: Call reidentifyText - const response: ReidentifyTextResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .reidentifyText(reidentifyRequest, options); - - // Step 7: Handle response - console.log('Re-identified Text Response:', response); - - } catch (error) { - // Comprehensive Error Handling - if (error instanceof SkyflowError) { - console.error('Skyflow Specific Error:', { - httpCode: error.error?.httpCode, - grpcCode: error.error?.grpcCode, - httpStatus: error.error?.httpStatus, - message: error.message, - details: error.error?.details, - }); - } else { - console.error('Unexpected Error:', JSON.stringify(error)); - } - } -} - -// Invoke the reidentify text function -performReidentifyText(); \ No newline at end of file diff --git a/samples/vault-api/insert-byot.ts b/samples/vault-api/insert-byot.ts index 020099d6..9824ff0e 100644 --- a/samples/vault-api/insert-byot.ts +++ b/samples/vault-api/insert-byot.ts @@ -9,9 +9,7 @@ import { VaultConfig, SkyflowConfig, InsertResponse, - SkyflowError, - TokenCredentials -} from 'skyflow-node'; + SkyflowError} from 'skyflow-node'; /** * Skyflow Insert with BYOT Example diff --git a/skyvault/README.md b/skyvault/README.md index 5f765097..55cb491f 100644 --- a/skyvault/README.md +++ b/skyvault/README.md @@ -47,10 +47,13 @@ Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow - [Retrieve Existing Tokens: `.tokenize(request)`](#retrieve-existing-tokens-tokenizerequest) - [Construct a `.tokenize()` request](#construct-a-tokenize-request) - [Detect](#detect) - - [De-identify Text: `.deidentifyText(request, options)`](#de-identify-text-deidentifytextrequest-options) - - [Re-identify Text: `.reidentifyText(request, options)`](#re-identify-text-reidentifytextrequest-options) - - [De-identify File: `.deidentifyFile(fileReq, options)`](#de-identify-file-deidentifyfilefilereq-options) - - [Get Run: `.getDetectRun(request)`](#get-run-getdetectrunrequest) + - [De-identify a string: `.deidentifyString(request)`](#de-identify-a-string-deidentifystringrequest) + - [Detect configuration](#detect-configuration) + - [De-identify a file: `.deidentifyFile(request)`](#de-identify-a-file-deidentifyfilerequest) + - [Re-identify a string: `.reidentifyString(request)`](#re-identify-a-string-reidentifystringrequest) + - [Re-identify a file: `.reidentifyFile(request)`](#re-identify-a-file-reidentifyfilerequest) + - [Get run: `.getRun(request)`](#get-run-getrunrequest) + - [Check guardrails: `.checkGuardrails(request)`](#check-guardrails-checkguardrailsrequest) - [Connections](#connections) - [Configure a connection](#configure-a-connection) - [Invoke a connection](#invoke-a-connection) @@ -627,213 +630,288 @@ Tokenization Result: { tokens: ['5391-4629-3722-7102', '0f6b8a2c-90ab-4cde-9def- ## Detect -De-identify and reidentify sensitive data in text and files using Skyflow Detect, which supports advanced privacy-preserving workflows. +Skyflow Detect finds and de-identifies sensitive data in free text and files, then re-identifies it on demand. The SDK exposes the Detect V2 API through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault and uses the vault's credentials and log level. -### De-identify Text: `.deidentifyText(request, options)` +Detect configurations are created in Studio or through the management API. The SDK does not create, update or delete them. -De-identify or anonymize text using the `deidentifyText` method. +| Method | Request | Response | Description | +|--------|---------|----------|-------------| +| `deidentifyString(request)` | [`DeidentifyStringRequest`](docs/api_reference.md#deidentifystringrequest) | [`DeidentifyStringResponse`](docs/api_reference.md#deidentifystringresponse) | Detect and replace sensitive data in a string | +| `deidentifyFile(request)` | [`DeidentifyFileRequest`](docs/api_reference.md#deidentifyfilerequest) | [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) | Submit a file for de-identification, optionally polling the run to completion | +| `reidentifyString(request)` | [`ReidentifyStringRequest`](docs/api_reference.md#reidentifystringrequest) | [`ReidentifyStringResponse`](docs/api_reference.md#reidentifystringresponse) | Replace tokens in a string with their original or redacted values | +| `reidentifyFile(request)` | [`ReidentifyFileRequest`](docs/api_reference.md#reidentifyfilerequest) | [`ReidentifyFileResponse`](docs/api_reference.md#reidentifyfileresponse) | Replace tokens in a file with their original or redacted values | +| `getRun(request)` | [`GetRunRequest`](docs/api_reference.md#getrunrequest) | [`GetRunResponse`](docs/api_reference.md#getrunresponse) | Retrieve the status and outputs of a Detect file run | +| `checkGuardrails(request)` | [`CheckGuardrailsRequest`](docs/api_reference.md#checkguardrailsrequest) | [`CheckGuardrailsResponse`](docs/api_reference.md#checkguardrailsresponse) | Screen text for toxicity and denied topics | -Create a de-identify text request with the [`DeidentifyTextRequest`](docs/api_reference.md#deidentifytextrequest) class, which includes the text to be deidentified. Provide optional parameters using the [`DeidentifyTextOptions`](docs/api_reference.md#deidentifytextoptions) class. +**Before you start** + +- **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy that grants the Detokenization action on the destination table and columns, unless it is a Vault Owner. +- **Configuration.** `configurationId` is the id of a Detect configuration created through the management API. The alternative is an inline `DetectConfiguration` passed as `configuration` (see [Detect configuration](#detect-configuration)). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings. Sending both is rejected by the API with HTTP 400. +- **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the `table.column` named by `destination`. That table and column must already exist in the vault. +- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](docs/api_reference.md#detect-api). + +### De-identify a string: `.deidentifyString(request)` + +`DeidentifyStringRequest` takes the text plus, optionally, `configurationId` (a stored Detect configuration) or `configuration` (an inline `DetectConfiguration`). The SDK fills the vault id into the configuration from the controller. ```typescript import { - DeidentifyTextRequest, - DeidentifyTextOptions, + DeidentifyStringRequest, + DeidentifyStringResponse, + DetectConfiguration, + EntityType, + DeidentificationType, + ReturnEntitiesType, SkyflowError, - TokenFormat, - TokenType, - Transformations, - DetectEntities, -} from "skyflow-node"; +} from 'skyflow-node'; -// Prepare the text to be deidentified -const deidentifyTextRequest = new DeidentifyTextRequest( - "", +// Stored configuration +const stored = new DeidentifyStringRequest( + 'My name is John Doe, and my email is johndoe@acme.com.', + { configurationId: '' }, ); - -// Configure DeidentifyTextOptions -const options = new DeidentifyTextOptions(); -options.setEntities([DetectEntities.ACCOUNT_NUMBER, DetectEntities.SSN]); // Entities to de-identify -options.setAllowRegexList([""]); // Allowlist regex patterns -options.setRestrictRegexList([""]); // Restrict regex patterns - -const tokenFormat = new TokenFormat(); -tokenFormat.setDefault(TokenType.VAULT_TOKEN); // default format -options.setTokenFormat(tokenFormat); - -const transformations = new Transformations(); -transformations.setShiftDays({ - max: 30, // maximum days to shift - min: 10, // minimum days to shift - entities: [DetectEntities.ACCOUNT_NUMBER, DetectEntities.SSN], // entity types to apply the shift -}); -options.setTransformations(transformations); - -// Call deidentifyText -const response = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyText(deidentifyTextRequest, options); - -console.log("De-identify Text Response:", response); +const response: DeidentifyStringResponse = await skyflowClient.detect().deidentifyString(stored); +console.log(response.processedText); +// My name is [NAME_1], and my email is [EMAIL_ADDRESS_1]. + +// Inline configuration +const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '
.', // table.column that stores the value + }, + { entityType: EntityType.ALL, deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER }, + ], + skip: ['Skyflow'], + returnEntities: ReturnEntitiesType.EXCLUDE_SENSITIVE_DATA, + }, +}; +const inline = new DeidentifyStringRequest( + 'My name is John Doe, and my email is johndoe@acme.com.', + { configuration }, +); +const inlineResponse = await skyflowClient.detect().deidentifyString(inline); +for (const entity of inlineResponse.entities) { + console.log(entity.entityType, entity.token, entity.location?.startIndex, entity.location?.endIndex); +} +console.log(inlineResponse.metrics?.wordCount); ``` -Returns a [`DeidentifyTextResponse`](docs/api_reference.md#deidentifytextresponse) (`processedText`, `entities`, `wordCount`, `charCount`, `errors`): - -```text -De-identify Text Response: { processedText: 'Hello, [PERSON_1]! My SSN is [SSN_1].', entities: [{ token: '[PERSON_1]', value: 'John', entity: 'NAME', scores: { NAME: 0.98 } }], wordCount: 8, charCount: 37, errors: null } -``` +`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. > [!TIP] -> See the full example in the samples directory: [deidentify-text.ts](samples/detect-api/deidentify-text.ts) +> See the full example in the samples directory: [deidentify-string.ts](samples/detect-api/deidentify-string.ts) -### Re-identify Text: `.reidentifyText(request, options)` +### Detect configuration -Re-identify text using the `reidentifyText` method. Create a reidentify text request with the [`ReidentifyTextRequest`](docs/api_reference.md#reidentifytextrequest) class, which includes the redacted or de-identified text to be re-identified. Provide optional parameters using the [`ReidentifyTextOptions`](docs/api_reference.md#reidentifytextoptions) class to control how specific entities are returned (as redacted, masked, or plain text). +`DetectConfiguration` is the inline form of a stored configuration and accepts the same options. `detect` applies to string and file inputs; `fileMapping` and `media` apply to file inputs only. All of these are plain TypeScript interfaces, so write them as object literals. + +| Type | Fields | Notes | +|------|--------|-------| +| `Detect` | `entities`, `objectEntities`, `restrict`, `skip`, `customTypes`, `returnEntities` | `restrict` and `skip` are regexes that are always, respectively never, treated as sensitive. `returnEntities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. | +| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` is used only with `VAULT_TOKEN`. | +| `Transformation` / `ShiftDates` | `shiftDates`; `minDays`, `maxDays` | Shifts detected dates by a random number of days within the range. | +| `CustomType` | `label`, `match` | A regex-defined entity type. `label` is reported as the entity type of its matches. | +| `ObjectEntity` | `entityType`, `deidentificationType` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. | +| `FileMapping` | `source`, `destination`, `outputFileNameSuffix`, `entitiesDestination`, `objectEntitiesDestination` | Where the processed file and the entity files are written. File inputs only. | +| `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. | +| `Audio` / `Bleep` | `outputProcessedAudio`, `outputTranscription`, `bleep`; `startPadding`, `stopPadding`, `frequency`, `gain` | `outputTranscription` is an `OutputTranscriptionType`. `Bleep` is the tone laid over detected speech; paddings are in seconds, frequency in Hz. | +| `Document` / `Pdf` | `pdf`; `processingMode`, `density`, `maxResolution` | `processingMode` is a `PdfProcessingMode` (`NONE`, `OCR` or `TEXT_LAYER`). `density` is the rendering DPI. | +| `Image` | `outputProcessedImage`, `outputOcrText`, `maskingMethod` | `maskingMethod` is a `MaskingMethod` (`NONE`, `BLUR` or `BLACKBOX`). | ```typescript import { - ReidentifyTextRequest, - ReidentifyTextOptions, - SkyflowError, - DetectEntities, - ReidentifyTextResponse, -} from "skyflow-node"; + DetectConfiguration, + EntityType, + DeidentificationType, + ObjectEntityType, + ObjectDeidentificationType, + MaskingMethod, + PdfProcessingMode, + OutputTranscriptionType, +} from 'skyflow-node'; + +const configuration: DetectConfiguration = { + detect: { + entities: [ + { + entityType: EntityType.DOB, + deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, + transformation: { shiftDates: { minDays: 10, maxDays: 30 } }, + }, + { entityType: EntityType.ALL }, + ], + objectEntities: [{ entityType: ObjectEntityType.FACE, deidentificationType: ObjectDeidentificationType.REDACT }], + customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], + restrict: ['[0-9]{3}-[0-9]{2}-[0-9]{4}'], + }, + media: { + image: { maskingMethod: MaskingMethod.BLUR, outputOcrText: true }, + document: { pdf: { processingMode: PdfProcessingMode.OCR } }, + audio: { + outputTranscription: OutputTranscriptionType.TRANSCRIPTION, + bleep: { startPadding: 0.5, stopPadding: 0.5 }, + }, + }, +}; +``` -// Prepare the redacted text to be re-identified -const request = new ReidentifyTextRequest(""); +### De-identify a file: `.deidentifyFile(request)` -// Configure ReidentifyTextOptions -const options = new ReidentifyTextOptions(); -options.setRedactedEntities([DetectEntities.SSN]); // Entities to keep redacted -options.setMaskedEntities([DetectEntities.CREDIT_CARD_NUMBER]); // Entities to mask -options.setPlainTextEntities([DetectEntities.NAME]); // Entities to return as plain text +File de-identification is asynchronous on the API: the submit call returns a run id and the result is retrieved from the run. `deidentifyFile` exposes both modes. -// Call reidentifyText -const response: ReidentifyTextResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .reidentifyText(request, options); +- **Submit only** (default): one call, returns `runId`. `status` is `null` and nothing is written to disk. Retrieve the result later with [`getRun`](#get-run-getrunrequest). +- **Submit and poll**: set `pollOptions`. The SDK polls the run with exponential backoff (1, 2, 4, 8, 16, then 32 seconds between lookups) until it is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `waitTime` (default 60 s, max 300) or `maxAttempts` (default 23, max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED` so you can resume with `getRun`. -console.log("Reidentify Text Response:", response); -``` +The file is supplied in one of three ways: a `dataSource` plus `value` (`BASE64` content, a `SKYFLOW_ID`, or a `PRESIGNED_URL`), a `file` (a web `File`, Node.js 20+), or a `filePath`. A local file is read, base64-encoded, and its `dataFormat` inferred from the extension. Supply exactly one of `value`, `file` or `filePath`. `configurationId` and `configuration` follow the same rules as for strings. -Returns a [`ReidentifyTextResponse`](docs/api_reference.md#reidentifytextresponse) (`processedText`): +`dataFormat` accepts every `FileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. -```text -Reidentify Text Response: { processedText: 'Hello, John! My SSN is 123-45-6789.' } +`outputDirectory` is meaningful only when polling. Setting it without `pollOptions` fails client-side validation before any request is sent. + +```typescript +import { DeidentifyFileRequest, DeidentifyFileResponse, DetectRunStatus } from 'skyflow-node'; + +// Submit only +const submitted: DeidentifyFileResponse = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ + filePath: '/data/intake/patient-notes.pdf', // sent as BASE64, format inferred as PDF + configurationId: '', + }), +); +const runId = submitted.runId; + +// Submit and poll +const response = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ + filePath: '/data/intake/patient-notes.pdf', + configurationId: '', + pollOptions: { waitTime: 120 }, + outputDirectory: '/data/processed', // optional + }), +); + +switch (response.status) { + case DetectRunStatus.SUCCESS: + // For local-file requests the outputs are already on disk: + // /data/processed/processed-patient-notes.pdf the redacted file + // /data/processed/processed-entities-patient-notes.json the detected entities + // response.output and response.metrics carry the same data. + break; + case DetectRunStatus.FAILED: + console.error(response.message); + break; + default: // IN_PROGRESS or QUEUED: budget ran out, resume with getRun(runId) + break; +} ``` +Outputs are written only when polling reached `SUCCESS`, the request was built from a local file, and the run delivered them as base64. A request built from a `SKYFLOW_ID` or `PRESIGNED_URL` has no local name to derive from, and presigned-URL outputs are not fetched, so in those cases read `output` and write the files yourself. Files are named `processed-.`, with the entities output as `processed-entities-.json`; `outputDirectory` is created if missing and defaults to the working directory. Writing outputs needs the Node.js file system and is not supported in Cloudflare Workers. + > [!TIP] -> See the full example in the samples directory: [reidentify-text.ts](samples/detect-api/reidentify-text.ts) +> See the full examples in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](samples/detect-api/deidentify-file-async.ts) -### De-identify File: `.deidentifyFile(fileReq, options)` +### Re-identify a string: `.reidentifyString(request)` -De-identify files using the `deidentifyFile` method. Create a de-identify file request with the [`DeidentifyFileRequest`](docs/api_reference.md#deidentifyfilerequest) class, which includes the file to be deidentified (such as images, PDFs, audio, documents, spreadsheets, or presentations). Provide optional parameters using the [`DeidentifyFileOptions`](docs/api_reference.md#deidentifyfileoptions) class to control how entities are detected and deidentified, as well as how the output is generated for different file types. +Re-identification looks tokens up in the vault, so it applies to values that were de-identified with `VAULT_TOKEN`. Placeholders produced by `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not stored anywhere and cannot be re-identified. -> [!NOTE] -> File de-identification requires Node.js v20.x or above. +`ReidentifyStringRequest` takes the tokenized text and an optional list of `RedactionLevel` rules. For skyvault vaults each rule names an `entityName` and the `redactionType` to render it with. Without rules every token is replaced by its original value. ```typescript -import { - DeidentifyFileRequest, - DeidentifyFileOptions, - DeidentifyFileResponse - SkyflowError, - DetectEntities, - MaskingMethod, - DetectOutputTranscription, - Bleep -} from 'skyflow-node'; +import { ReidentifyStringRequest, ReidentifyStringResponse, EntityType, RedactionType } from 'skyflow-node'; -// Prepare the file to be deidentified -const filePath: string = ''; -const buffer = fs.readFileSync(filePath); -const file = new File([buffer], filePath); +const response: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest('My name is [NAME_1].', { + redactionLevel: [{ entityName: EntityType.NAME, redactionType: RedactionType.MASKED }], + }), +); +console.log(response.processedText); +// My name is J**n D*e. +``` -// Construct the file input by providing either a file object or file path, but not both -const fileInput: FileInput = { file: file } // OR const fileInput: FileInput = { filePath: filePath } -const fileReq = new DeidentifyFileRequest(fileInput); +`RedactionLevel` is the skyvault SDK's own type and has only `entityName` and `redactionType`; the `tokenGroupName` and `redactionPattern` rules of the flowvault SDK do not exist here. Both fields are optional on the API, so the SDK passes the list through as given and the server rejects invalid combinations. -// Configure DeidentifyFileOptions -const options = new DeidentifyFileOptions(); -options.setEntities([DetectEntities.SSN, DetectEntities.ACCOUNT_NUMBER]); +> [!TIP] +> See the full example in the samples directory: [reidentify-string.ts](samples/detect-api/reidentify-string.ts) -const tokenFormat = new TokenFormat(); // Token format for deidentified entities -tokenFormat.setDefault(TokenType.ENTITY_ONLY); -options.setTokenFormat(tokenFormat); +### Re-identify a file: `.reidentifyFile(request)` +`reidentifyFile` is synchronous: the re-identified file comes back in the response, not through a run id. `ReidentifyFileRequest` takes a `dataSource` (`ReidentifySourceType.BASE64` or `PRESIGNED_URL`), the matching `value`, an optional `dataFormat` and the same optional `RedactionLevel` rules as `reidentifyString`. `dataFormat` is limited to the formats that support `VAULT_TOKEN`: `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. -options.setOutputDirectory(''); // Output directory for saving the deidentified file. Not supported in Cloudflare workers. -options.setWaitTime(64); // Wait time for polling (max 64 seconds; returns runId + status if exceeded) +```typescript +import * as fs from 'fs'; +import { ReidentifyFileRequest, ReidentifySourceType, FileDataFormat, DetectRunStatus } from 'skyflow-node'; +const content = fs.readFileSync('tokenized.txt').toString('base64'); -tifyFile -const response: DeidentifyFileResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .deidentifyFile(fileReq, options); +const response = await skyflowClient.detect().reidentifyFile( + new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: FileDataFormat.TXT }), +); -console.log('De-identify File Response:', response); +if (response.status === DetectRunStatus.SUCCESS) { + for (const output of response.output) { + // output.processedFile is base64 or a presigned URL, per response.outputType + const processed = Buffer.from(output.processedFile ?? '', 'base64'); + } +} ``` -Returns a [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) (`fileBase64`, `file`, `runId`, `status`, `errors`). If processing exceeds `setWaitTime`, only `runId` and `status` are populated — poll with `getDetectRun`: - -```text -De-identify File Response: { runId: 'run-abc123', status: 'IN_PROGRESS', fileBase64: undefined, errors: null } -``` +`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. -**Supported file types:** +> [!TIP] +> See the full example in the samples directory: [reidentify-file.ts](samples/detect-api/reidentify-file.ts) -- Documents: `doc`, `docx`, `pdf` -- PDFs: `pdf` -- Images: `bmp`, `jpeg`, `jpg`, `png`, `tif`, `tiff` -- Structured text: `json`, `xml` -- Spreadsheets: `csv`, `xls`, `xlsx` -- Presentations: `ppt`, `pptx` -- Audio: `mp3`, `wav` +### Get run: `.getRun(request)` -**Notes:** -- Transformations can't be applied to Documents, Images, or PDFs. -- `setWaitTime` must be ≤ 64 seconds; otherwise, an error is thrown. -- If the API takes more than `waitTime` seconds, the response contains only `runId` and `status`. Use `.getDetectRun()` to poll for the result. +File de-identification is asynchronous: the submit call returns a run id, and `getRun` retrieves the run's status and, once it has succeeded, its outputs. `output` is empty and `outputType` and `metrics` are `null` until the run completes. -For all `DeidentifyFileOptions` setters and the `Bleep` class, see [DeidentifyFileOptions](docs/api_reference.md#deidentifyfileoptions) in the API reference. +```typescript +import { GetRunRequest, GetRunResponse, DetectRunStatus } from 'skyflow-node'; + +const run: GetRunResponse = await skyflowClient.detect().getRun(new GetRunRequest('')); + +switch (run.status) { + case DetectRunStatus.SUCCESS: + for (const output of run.output) { + // output.processedFile is base64 or a presigned URL, per run.outputType + // output.processedFileType is e.g. REDACTED_FILE or ENTITIES + // output.processedFileExtension is a FileDataFormat such as PDF + } + break; + case DetectRunStatus.FAILED: + console.error(run.message); + break; + default: // QUEUED, IN_PROGRESS, UNKNOWN: poll again later + break; +} +``` > [!TIP] -> See the full example in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) +> See the full example in the samples directory: [get-run.ts](samples/detect-api/get-run.ts) -### Get Run: `.getDetectRun(request)` +### Check guardrails: `.checkGuardrails(request)` -Retrieve the results of a previously started file de-identification operation (or 'run') using the `getDetectRun(...)` method. Initialize the [`GetDetectRunRequest`](docs/api_reference.md#getdetectrunrequest) class with the `runId` returned from a prior `.deidentifyFile(fileReq, options)` call. Fetch the final results of the file de-identification operation once they are available. +`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. ```typescript -import { - GetDetectRunRequest, - DeidentifyFileResponse, - DeidentifyFileResponse - SkyflowError -} from 'skyflow-node'; +import { CheckGuardrailsRequest, GuardrailsValidation } from 'skyflow-node'; -// Prepare the GetDetectRunRequest with the runId from a previous deidentifyFile call -const request = new GetDetectRunRequest({ - runId: '', // Replace with the runId you received earlier -}); - -// Step 2: Call getDetectRun -const response: DeidentifyFileResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .getDetectRun(request); +const response = await skyflowClient.detect().checkGuardrails( + new CheckGuardrailsRequest(userPrompt, { checkToxicity: true, denyTopics: ['politics', 'medical advice'] }), +); -// Step 3: Handle the response -console.log('Get Detect Run Response:', response); +if (response.validation === GuardrailsValidation.FAILED) { + throw new Error(`blocked: toxic=${response.toxic} deniedTopic=${response.deniedTopic}`); +} ``` -Returns a [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) with the current `status` (and the processed file once complete): - -```text -Get Detect Run Response: { status: 'SUCCESS', runId: 'run-abc123', fileBase64: '', errors: null } -``` +All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling). Client-side validation fails before any network call. > [!TIP] -> See the full example in the samples directory: [get-detect-run.ts](samples/detect-api/get-detect-run.ts) +> See the full example in the samples directory: [check-guardrails.ts](samples/detect-api/check-guardrails.ts) ## Connections @@ -1307,7 +1385,11 @@ import { // Request item shapes DetokenizeData, // { token: string; redactionType?: RedactionType } TokenizeRequestType, // { value: string; columnGroup: string } - FileInput, // { file: File } | { filePath: string } + + // Detect configuration and rule shapes + DetectConfiguration, // inline Detect configuration: { detect?, fileMapping?, media? } + RedactionLevel, // { entityName?, redactionType? } (skyvault); the flowvault SDK has { tokenGroupName?, redactionPattern? } + PollOptions, // { waitTime?: number; maxAttempts?: number } // Response record shapes InsertResponseType, // { skyflowId: string; [field: string]: unknown } diff --git a/skyvault/src/index.ts b/skyvault/src/index.ts index a49d2ad2..7dc73ee8 100644 --- a/skyvault/src/index.ts +++ b/skyvault/src/index.ts @@ -1,5 +1,5 @@ import Skyflow from './vault/skyflow'; -import { LogLevel, Env, RedactionType, RequestMethod, OrderByEnum, TokenMode, DetectEntities } from "skyflow-common"; +import { LogLevel, Env, RedactionType, RequestMethod, OrderByEnum, TokenMode } from "skyflow-common"; import InsertRequest from './vault/model/request/insert'; import InsertOptions from './vault/model/options/insert'; import GetRequest from './vault/model/request/get'; @@ -28,25 +28,68 @@ import UpdateResponse from './vault/model/response/update'; import FileUploadResponse from './vault/model/response/file-upload'; import QueryResponse from './vault/model/response/query'; import InvokeConnectionResponse from './vault/model/response/invoke/invoke'; -import { SkyflowConfig, TokenizeRequestType, DetokenizeData, InsertResponseType, GetResponseData, QueryResponseType, IndexRange, FileInput, StringKeyValueMapType } from './vault/types'; +import { SkyflowConfig, TokenizeRequestType, DetokenizeData, InsertResponseType, GetResponseData, QueryResponseType, StringKeyValueMapType } from './vault/types'; import { VaultConfig } from "skyflow-common"; import { SkyflowError } from "skyflow-common"; import { ConnectionConfig } from "skyflow-common"; -import DeidentifyTextRequest from './vault/model/request/deidentify-text'; -import DeidentifyTextOptions from './vault/model/options/deidentify-text'; -import DeidentifyTextResponse from './vault/model/response/deidentify-text'; -import ReidentifyTextRequest from './vault/model/request/reidentify-text'; -import ReidentifyTextOptions from './vault/model/options/reidentify-text'; -import ReidentifyTextResponse from './vault/model/response/reidentify-text'; -import TokenFormat from './vault/model/options/deidentify-text/token-format'; -import Transformations from './vault/model/options/deidentify-text/transformations'; -import DeidentifyFileOptions from './vault/model/options/deidentify-file'; -import DeidentifyFileRequest from './vault/model/request/deidentify-file'; -import DeidentifyFileResponse from './vault/model/response/deidentify-file'; -import GetDetectRunRequest from './vault/model/request/get-detect-run'; -import { TokenType, MaskingMethod, DetectOutputTranscription } from "skyflow-common"; -import { Bleep } from './vault/model/options/deidentify-file/bleep-audio'; +// Detect V2 (shared with the flowvault SDK through skyflow-common). +import { + DeidentifyStringRequest, + DeidentifyStringResponse, + DeidentifyFileRequest, + DeidentifyFileResponse, + ReidentifyStringRequest, + ReidentifyStringResponse, + ReidentifyFileRequest, + ReidentifyFileResponse, + GetRunRequest, + GetRunResponse, + CheckGuardrailsRequest, + CheckGuardrailsResponse, + EntityType, + DeidentificationType, + FileDataFormat, + DetectRunStatus, + ReturnEntitiesType, + MaskingMethod, + OutputTranscriptionType, + PdfProcessingMode, + ObjectEntityType, + ObjectDeidentificationType, + GuardrailsValidation, + POLL_OPTIONS, +} from "skyflow-common"; +import type { + DeidentifyStringRequestFields, + DeidentifyFileRequestFields, + ReidentifyStringRequestFields, + ReidentifyFileRequestFields, + CheckGuardrailsRequestFields, + DetectRunFields, + DetectConfiguration, + Detect, + Entity, + Transformation, + ShiftDates, + CustomType, + ObjectEntity, + FileMapping, + Media, + Audio, + Bleep, + Document, + Pdf, + Image, + PollOptions, + DetectedEntity, + EntityLocation, + Metrics, + FileOutput, +} from "skyflow-common"; import { SkyflowRecordError } from "skyflow-common"; +// Detect V2 types whose values depend on the vault type: the skyvault versions. +import { DeidentifySourceType, ReidentifySourceType } from "./vault/detect/types"; +import type { RedactionLevel } from "./vault/detect/types"; export { Env, LogLevel, @@ -98,28 +141,62 @@ export { TokenizeResponse, InvokeConnectionRequest, InvokeConnectionResponse, - DeidentifyTextRequest, - DeidentifyTextOptions, - DetectEntities, - DeidentifyTextResponse, - ReidentifyTextRequest, - ReidentifyTextOptions, - ReidentifyTextResponse, - TokenFormat, - Transformations, - DeidentifyFileOptions, + // Detect V2 + DeidentifyStringRequest, + DeidentifyStringRequestFields, + DeidentifyStringResponse, DeidentifyFileRequest, + DeidentifyFileRequestFields, DeidentifyFileResponse, - TokenType, + ReidentifyStringRequest, + ReidentifyStringRequestFields, + ReidentifyStringResponse, + ReidentifyFileRequest, + ReidentifyFileRequestFields, + ReidentifyFileResponse, + GetRunRequest, + GetRunResponse, + DetectRunFields, + CheckGuardrailsRequest, + CheckGuardrailsRequestFields, + CheckGuardrailsResponse, + DetectConfiguration, + Detect, + Entity, + Transformation, + ShiftDates, + CustomType, + ObjectEntity, + FileMapping, + Media, + Audio, Bleep, + Document, + Pdf, + Image, + RedactionLevel, + PollOptions, + POLL_OPTIONS, + DetectedEntity, + EntityLocation, + Metrics, + FileOutput, + EntityType, + DeidentificationType, + DeidentifySourceType, + ReidentifySourceType, + FileDataFormat, + DetectRunStatus, + ReturnEntitiesType, MaskingMethod, - DetectOutputTranscription, - GetDetectRunRequest, + OutputTranscriptionType, + PdfProcessingMode, + ObjectEntityType, + ObjectDeidentificationType, + GuardrailsValidation, SkyflowRecordError, InsertResponseType, GetResponseData, QueryResponseType, - IndexRange, StringKeyValueMapType, - FileInput }; \ No newline at end of file diff --git a/skyvault/src/utils/validations/index.ts b/skyvault/src/utils/validations/index.ts index f1ae580e..93c91c7f 100644 --- a/skyvault/src/utils/validations/index.ts +++ b/skyvault/src/utils/validations/index.ts @@ -17,16 +17,6 @@ import UpdateRequest from "../../vault/model/request/update"; import * as fs from 'fs'; import * as path from 'path'; import FileUploadOptions from "../../vault/model/options/fileUpload"; -import DeidentifyTextRequest from "../../vault/model/request/deidentify-text"; -import DeidentifyTextOptions from "../../vault/model/options/deidentify-text"; -import TokenFormat from "../../vault/model/options/deidentify-text/token-format"; -import Transformations from "../../vault/model/options/deidentify-text/transformations"; -import ReidentifyTextRequest from "../../vault/model/request/reidentify-text"; -import ReidentifyTextOptions from "../../vault/model/options/reidentify-text"; -import DeidentifyFileOptions from "../../vault/model/options/deidentify-file"; -import DeidentifyFileRequest from "../../vault/model/request/deidentify-file"; -import { Bleep } from "../../vault/model/options/deidentify-file/bleep-audio"; -import GetDetectRunRequest from "../../vault/model/request/get-detect-run"; export function isRedactionType(value?: string): boolean { return value !== undefined && Object.values(RedactionType).includes(value as RedactionType); @@ -660,199 +650,6 @@ export const validateQueryRequest = (queryRequest: QueryRequest, logLevel: LogLe } } -export const validateDeIdentifyTextRequest = (deIdentifyTextRequest: DeidentifyTextRequest, options?: DeidentifyTextOptions, logLevel: LogLevel = LogLevel.ERROR) => { - if (!deIdentifyTextRequest.text || typeof deIdentifyTextRequest.text !== 'string' || deIdentifyTextRequest.text.trim().length === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_DEIDENTIFY); - } - - if (options) { - if (options.getEntities() && !Array.isArray(options.getEntities())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ENTITIES_IN_DEIDENTIFY); - } - - if (options.getAllowRegexList() && !Array.isArray(options.getAllowRegexList())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ALLOW_REGEX_LIST); - } - - if (options.getRestrictRegexList() && !Array.isArray(options.getRestrictRegexList())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_RESTRICT_REGEX_LIST); - } - - if (options.getTokenFormat() && !(options.getTokenFormat() instanceof TokenFormat)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_FORMAT); - } - - if (options.getTransformations() && !(options.getTransformations() instanceof Transformations)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TRANSFORMATIONS); - } - } -}; - -export const validateReidentifyTextRequest = (request: ReidentifyTextRequest, options?: ReidentifyTextOptions, logLevel: LogLevel = LogLevel.ERROR) => { - if (!request.text || typeof request.text !== 'string' || request.text.trim().length === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_REIDENTIFY); - } - - if (options) { - if (options.getRedactedEntities() && !Array.isArray(options.getRedactedEntities())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_REDACTED_ENTITIES_IN_REIDENTIFY); - } - - if (options.getMaskedEntities() && !Array.isArray(options.getMaskedEntities())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_MASKED_ENTITIES_IN_REIDENTIFY); - } - if (options.getPlainTextEntities() && !Array.isArray(options.getPlainTextEntities())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_PLAIN_TEXT_ENTITIES_IN_REIDENTIFY); - } - } -}; - -export const validateDeidentifyFileRequest = (deidentifyFileRequest: DeidentifyFileRequest, deidentifyFileOptions?: DeidentifyFileOptions, logLevel: LogLevel = LogLevel.ERROR) => { - if (!deidentifyFileRequest) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); - } - - const fileType = deidentifyFileRequest.getFile(); - - const hasFile = 'file' in fileType && fileType.file; - const hasFilePath = 'filePath' in fileType && fileType.filePath; - - // Must provide exactly one of file or filePath - if ((hasFile && hasFilePath) || (!hasFile && !hasFilePath)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_INPUT); - } - - if (hasFile) { - const file = fileType.file; - if (!(file instanceof File)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - } - if (!file.name || typeof file.name !== 'string' || file.name.trim().length === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - } - const fileBaseName = path.parse(file.name).name; - if (!fileBaseName || typeof fileBaseName !== 'string' || fileBaseName.trim().length === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - } - } else if (hasFilePath) { - const filePath = fileType.filePath; - if (typeof filePath !== 'string' || filePath.trim().length === 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_PATH); - } - if (!fs.existsSync(filePath) || !fs.lstatSync(filePath).isFile()) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_PATH); - } - } - - // Validate options if provided - if (deidentifyFileOptions) { - validateDeidentifyFileOptions(deidentifyFileOptions); - } -}; - -export const validateGetDetectRunRequest = (getDetectRunRequest: GetDetectRunRequest, logLevel: LogLevel = LogLevel.ERROR) => { - if (getDetectRunRequest) { - if (!getDetectRunRequest?.runId) { - printLog(logs.errorLogs.EMPTY_RUN_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.EMPTY_RUN_ID); - } - if (typeof getDetectRunRequest.runId !== 'string' || getDetectRunRequest.runId.trim().length === 0) { - printLog(logs.errorLogs.INVALID_RUN_ID, MessageType.ERROR, logLevel); - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_RUN_ID); - } - } -} - -export const validateDeidentifyFileOptions = (deidentifyFileOptions: DeidentifyFileOptions) => { - if (!deidentifyFileOptions) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_OPTIONS); - } - - //Validate waitTime - const waitTime = deidentifyFileOptions.getWaitTime(); - if (waitTime !== undefined) { - if (typeof waitTime !== 'number' || waitTime > 64) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_WAIT_TIME); - } - } - - // Validate outputDirectory - const outputDirectory = deidentifyFileOptions.getOutputDirectory(); - if (outputDirectory !== undefined) { - if (typeof outputDirectory !== 'string') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_DIRECTORY); - } - if (!fs.existsSync(outputDirectory) || !fs.lstatSync(outputDirectory).isDirectory()) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_DIRECTORY_PATH); - } - } - - // Validate entities - if (deidentifyFileOptions.getEntities() !== undefined && (!Array.isArray(deidentifyFileOptions.getEntities()))) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ENTITIES); - } - - // Validate allowRegexList - if (deidentifyFileOptions.getAllowRegexList() && !Array.isArray(deidentifyFileOptions.getAllowRegexList())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_ALLOW_REGEX_LIST); - } - - // Validate restrictRegexList - if (deidentifyFileOptions.getRestrictRegexList() && !Array.isArray(deidentifyFileOptions.getRestrictRegexList())) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_RESTRICT_REGEX_LIST); - } - - // Validate tokenFormat - if (deidentifyFileOptions.getTokenFormat() && !(deidentifyFileOptions.getTokenFormat() instanceof TokenFormat)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TOKEN_FORMAT); - } - - const tokenFormat = deidentifyFileOptions.getTokenFormat(); - if(tokenFormat != null && tokenFormat.getVaultToken() != null && tokenFormat.getVaultToken()!.length > 0) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.TOKEN_FORMAT_NOT_ALLOWED); - } - - // Validate transformations - if (deidentifyFileOptions.getTransformations() && !(deidentifyFileOptions.getTransformations() instanceof Transformations)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_TRANSFORMATIONS); - } - - // Validate image-specific options - if (deidentifyFileOptions.getOutputProcessedImage() !== undefined && typeof deidentifyFileOptions.getOutputProcessedImage() !== 'boolean') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_PROCESSED_IMAGE); - } - - if (deidentifyFileOptions.getOutputOcrText() !== undefined && typeof deidentifyFileOptions.getOutputOcrText() !== 'boolean') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_OCR_TEXT); - } - - if (deidentifyFileOptions.getMaskingMethod() !== undefined && typeof deidentifyFileOptions.getMaskingMethod() !== 'string') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_MASKING_METHOD); - } - - // Validate PDF-specific options - if (deidentifyFileOptions.getPixelDensity() !== undefined && typeof deidentifyFileOptions.getPixelDensity() !== 'number') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_PIXEL_DENSITY); - } - - if (deidentifyFileOptions.getMaxResolution() !== undefined && typeof deidentifyFileOptions.getMaxResolution() !== 'number') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_MAX_RESOLUTION); - } - - // Validate audio-specific options - if (deidentifyFileOptions.getOutputProcessedAudio() !== undefined && typeof deidentifyFileOptions.getOutputProcessedAudio() !== 'boolean') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_PROCESSED_AUDIO); - } - - if (deidentifyFileOptions.getOutputTranscription() !== undefined && typeof deidentifyFileOptions.getOutputTranscription() !== 'string') { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_TRANSCRIPTION); - } - - if (deidentifyFileOptions.getBleep() && !(deidentifyFileOptions.getBleep() instanceof Bleep)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_BLEEP); - } -}; - function isStringKeyValueMap(obj: unknown): obj is StringKeyValueMapType { if (typeof obj !== 'object' || obj === null || Array.isArray(obj)) { return false; diff --git a/skyvault/src/vault/client/index.ts b/skyvault/src/vault/client/index.ts index 7bc5946d..f5ffc93a 100644 --- a/skyvault/src/vault/client/index.ts +++ b/skyvault/src/vault/client/index.ts @@ -1,12 +1,10 @@ -// imports -import { Files } from "../../ _generated_/rest/api/resources/files/client/Client"; +// imports import { Query } from "../../ _generated_/rest/api/resources/query/client/Client"; import { Records } from "../../ _generated_/rest/api/resources/records/client/Client"; -import { Strings } from "../../ _generated_/rest/api/resources/strings/client/Client"; import { Tokens } from "../../ _generated_/rest/api/resources/tokens/client/Client"; import { SkyflowError } from "skyflow-common"; import { errorMessages } from "skyflow-common"; -import { AuthInfo, AuthType, LogLevel, MessageType, printLog, TYPES, HTTP_HEADER, CONTENT_TYPE, BOOLEAN_STRING, HTTP_STATUS_CODE } from "skyflow-common"; +import { AuthInfo, AuthType, LogLevel, MessageType, printLog, TYPES, HTTP_HEADER, CONTENT_TYPE, BOOLEAN_STRING, HTTP_STATUS_CODE, getBearerToken } from "skyflow-common"; import { isExpired } from "skyflow-common"; import { logs } from "skyflow-common"; import { Credentials } from "skyflow-common"; @@ -26,10 +24,6 @@ class VaultClient { queryAPI!: Query; - stringsAPI!: Strings; - - filesAPI!: Files; - individualCredentials?: Credentials; skyflowCredentials?: Credentials; @@ -91,18 +85,6 @@ class VaultClient { case TYPES.QUERY: if (!this.queryAPI) { this.queryAPI = new Query(this.supplierOptions()); - } - break; - case TYPES.DEIDENTIFY_TEXT: - case TYPES.REIDENTIFY_TEXT: - if (!this.stringsAPI) { - this.stringsAPI = new Strings(this.supplierOptions()); - } - break; - case TYPES.DEIDENTIFY_FILE: - case TYPES.DETECT_RUN: - if (!this.filesAPI) { - this.filesAPI = new Files(this.supplierOptions()); } break; default: @@ -110,6 +92,17 @@ class VaultClient { } } + /** + * Resolves the bearer token or API key for the next call, reusing the cached + * one while it is valid and regenerating it otherwise. Used by the shared + * Detect controller, which builds its own generated clients. + */ + async resolveAuthToken(): Promise { + const authInfo = await getBearerToken(this.getCredentials(), this.logLevel); + this.initConfig(authInfo); + return authInfo.key; + } + getCredentials(): Credentials | undefined { if (this.authInfo?.key && !this.updateTriggered) { switch (this.authInfo.type) { diff --git a/skyvault/src/vault/controller/detect/index.ts b/skyvault/src/vault/controller/detect/index.ts index b1425a6f..ed3e6582 100644 --- a/skyvault/src/vault/controller/detect/index.ts +++ b/skyvault/src/vault/controller/detect/index.ts @@ -1,738 +1,32 @@ -//imports - -import path from "path"; -import { DeidentifyTextRequest as DeidentifyTextRequest2,DeidentifyAudioRequest, DeidentifyDocumentRequest, DeidentifyDocumentRequestFileDataFormat, DeidentifyFileRequestFileDataFormat, DeidentifyImageRequest, DeidentifyImageRequestFileDataFormat, DeidentifyImageRequestMaskingMethod, DeidentifyPdfRequest, DeidentifyPresentationRequest, DeidentifyPresentationRequestFileDataFormat, DeidentifySpreadsheetRequest, DeidentifySpreadsheetRequestFileDataFormat, DeidentifyStructuredTextRequest, DeidentifyStructuredTextRequestFileDataFormat, DetectedEntity, EntityType, GetRunRequest, Transformations as GeneratedTransformations, TokenTypeWithoutVault, DeidentifyStringResponse, DeidentifyStatusResponse, DeidentifyAudioRequestFileDataFormat } from "../../../ _generated_/rest/api"; -import { DeidentifyFileRequest as DeidentifyFileRequest2} from "../../../ _generated_/rest/api"; - -import { TokenType } from "../../../ _generated_/rest/api"; -import { DeidenitfyFileRequestTypes, generateSDKMetrics, getBearerToken, MessageType, parameterizedString, printLog, removeSDKVersion, SDK, TYPES, HTTP_HEADER, DETECT_STATUS, FILE_EXTENSION, FILE_FORMAT_TYPE, FILE_PROCESSING, ENCODING_TYPE } from "skyflow-common"; -import { logs } from "skyflow-common"; -import { validateDeIdentifyTextRequest, validateReidentifyTextRequest, validateDeidentifyFileRequest, validateGetDetectRunRequest } from "../../../utils/validations"; +import { BaseDetectController } from "skyflow-common"; import VaultClient from "../../client"; -import DeidentifyTextOptions from "../../model/options/deidentify-text"; -import ReidentifyTextOptions from "../../model/options/reidentify-text"; -import DeidentifyTextRequest from "../../model/request/deidentify-text"; -import ReidentifyTextRequest from "../../model/request/reidentify-text"; -import DeidentifiedTextResponse from "../../model/response/deidentify-text"; -import DeidentifyTextResponse from "../../model/response/deidentify-text"; -import ReidentifyTextResponse from "../../model/response/reidentify-text"; -import DeidentifyFileOptions from "../../model/options/deidentify-file"; -import DeidentifyFileRequest from "../../model/request/deidentify-file"; -import DeidentifyFileResponse from "../../model/response/deidentify-file"; -import * as fs from 'fs'; -import { SkyflowError } from "skyflow-common"; -import { SKYFLOW_ERROR_CODE } from "skyflow-common"; -import GetDetectRunRequest from "../../model/request/get-detect-run"; -import Transformations from "../../model/options/deidentify-text/transformations"; -import { SkyflowAllError } from "../../types"; -import { DeidentifyFileDetectRunResponse, DeidentifyFileOutput, DetectTextResponse, DetectFileResponse } from "../../types"; - -class DetectController { - - private client: VaultClient; - +import { DeidentifySourceType, ReidentifySourceType } from "../../detect/types"; + +/** + * Skyflow Detect operations for a skyvault vault, returned by `Skyflow.detect()` + * and `Skyflow.detect(vaultId)`. Every Detect V2 method (`deidentifyString`, + * `deidentifyFile`, `reidentifyString`, `reidentifyFile`, `getRun`, + * `checkGuardrails`) is inherited from the shared `BaseDetectController` in + * `skyflow-common`; the skyvault `VaultClient` supplies the vault URL, vault id, + * credentials and log level. + * + * This layer owns the vault-specific types: `deidentifyFile` takes the skyvault + * `DeidentifySourceType` (`BASE64`, `SKYFLOW_ID` or `PRESIGNED_URL`), and a run's or + * re-identification's `outputType` is resolved to the same enums, so a value the skyvault SDK + * does not offer comes back as `null`. A `VAULT_TOKEN` entity's `destination` is a + * `table.column` pair, and `RedactionLevel` rules use `entityName` and `redactionType`. + */ +class DetectController extends BaseDetectController { constructor(client: VaultClient) { - this.client = client; - } - - private createSdkHeaders() { - return { [SDK.METRICS_HEADER_KEY]: JSON.stringify(generateSDKMetrics()) }; - } - - private async getFileFromRequest(request: DeidentifyFileRequest): Promise { - const fileType = request.getFile(); - - if ('file' in fileType && fileType.file) { - // Already a File or Buffer - return fileType.file as File; - } else if ('filePath' in fileType && fileType.filePath) { - const filePath = fileType.filePath; - const buffer = await fs.promises.readFile(filePath); - return new File([new Uint8Array(buffer)], filePath); - } - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); - } - - private async getBase64FileContent(file: File){ - const arrayBuffer = await file.arrayBuffer(); - const buffer = Buffer.from(arrayBuffer); - const base64String = buffer.toString(ENCODING_TYPE.BASE64); - return base64String; - } - - private getTransformations(options?: DeidentifyFileOptions | DeidentifyTextOptions) { - const transformations = options?.getTransformations() as Transformations | undefined; - return { - shift_dates: { - max_days: transformations?.getShiftDays()?.max, - min_days: transformations?.getShiftDays()?.min, - entity_types: transformations?.getShiftDays()?.entities, - } - }; - } - - private async buildAudioRequest(baseRequest: File, options?: DeidentifyFileOptions, fileExtension?: string): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var audioRequest : DeidentifyAudioRequest = { - file: { - base64: base64String as string, - data_format: fileExtension as DeidentifyAudioRequestFileDataFormat, - }, - vault_id: this.client.vaultId, - entity_types: options?.getEntities() as EntityType[], - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - transformations: this.getTransformations(options) as GeneratedTransformations, - output_transcription: options?.getOutputTranscription(), - output_processed_audio: options?.getOutputProcessedAudio(), - bleep_gain: options?.getBleep()?.getGain(), - bleep_frequency: options?.getBleep()?.getFrequency(), - bleep_start_padding: options?.getBleep()?.getStartPadding(), - bleep_stop_padding: options?.getBleep()?.getStopPadding(), - } - return audioRequest; - } - private async buildTextFileRequest(baseRequest: File, options?: DeidentifyFileOptions): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - - var textFileRequest: DeidentifyTextRequest2 = { - vault_id: this.client.vaultId, - file: { - base64: base64String, - data_format: FILE_FORMAT_TYPE.TXT, - }, - entity_types: options?.getEntities() as EntityType[], - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - transformations: this.getTransformations(options) as GeneratedTransformations, - } - return textFileRequest; - } - private async buildPdfRequest(baseRequest: File, options?: DeidentifyFileOptions): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var pdfRequest: DeidentifyPdfRequest = { - file: { - base64: base64String as string, - data_format: FILE_FORMAT_TYPE.PDF, - }, - vault_id: this.client.vaultId, - entity_types: options?.getEntities() as EntityType[], - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - max_resolution: options?.getMaxResolution(), - density: options?.getPixelDensity(), - } - return pdfRequest; - } - - private async buildImageRequest(baseRequest: File, options?: DeidentifyFileOptions, fileExtension?: string): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var imageRequest: DeidentifyImageRequest = { - vault_id: this.client.vaultId, - file: { - base64: base64String as string, - data_format: fileExtension as DeidentifyImageRequestFileDataFormat, - }, - allow_regex: options?.getAllowRegexList(), - entity_types: options?.getEntities() as EntityType[], - masking_method: options?.getMaskingMethod() as DeidentifyImageRequestMaskingMethod, - output_ocr_text: options?.getOutputOcrText(), - output_processed_image: options?.getOutputProcessedImage(), - restrict_regex: options?.getRestrictRegexList(), - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - }; - - return imageRequest; - } - - private async buildPptRequest(baseRequest: File, options?: DeidentifyFileOptions, fileExtension?: string): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var pptRequest: DeidentifyPresentationRequest = { - vault_id: this.client.vaultId, - file: { - base64: base64String as string, - data_format: fileExtension as DeidentifyPresentationRequestFileDataFormat, - }, - entity_types: options?.getEntities() as EntityType[], - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - }; - return pptRequest; - } - - private async buildSpreadsheetRequest(baseRequest: File, options?: DeidentifyFileOptions, fileExtension?: string): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var spreadsheetRequest: DeidentifySpreadsheetRequest = { - vault_id: this.client.vaultId, - file: { - base64: base64String as string, - data_format: fileExtension as DeidentifySpreadsheetRequestFileDataFormat, - }, - entity_types: options?.getEntities() as EntityType[], - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - }; - return spreadsheetRequest; - } - - private async buildStructuredTextRequest(baseRequest: File, options?: DeidentifyFileOptions, fileExtension?: string): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var structuredTextRequest: DeidentifyStructuredTextRequest = { - vault_id: this.client.vaultId, - file: { - base64: base64String as string, - data_format: fileExtension as DeidentifyStructuredTextRequestFileDataFormat, - }, - entity_types: options?.getEntities() as EntityType[], - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - transformations: this.getTransformations(options) as GeneratedTransformations, - }; - return structuredTextRequest; - } - - private async buildDocumentRequest(baseRequest: File, options?: DeidentifyFileOptions, fileExtension?: string): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var documentRequest: DeidentifyDocumentRequest = { - vault_id: this.client.vaultId, - file: { - base64: base64String as string, - data_format: fileExtension as DeidentifyDocumentRequestFileDataFormat, - }, - entity_types: options?.getEntities() as EntityType[], - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - }; - return documentRequest; - } - - private async buildGenericFileRequest(baseRequest: File, options?: DeidentifyFileOptions, fileExtension?: string): Promise { - const base64String = await this.getBase64FileContent(baseRequest); - var genericRequest: DeidentifyFileRequest2 = { - vault_id: this.client.vaultId, - file: { - base64: base64String as string, - data_format: fileExtension as DeidentifyAudioRequestFileDataFormat, - }, - token_type: { - default: options?.getTokenFormat()?.getDefault(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenTypeWithoutVault, - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - transformations: this.getTransformations(options) as GeneratedTransformations, - entity_types: options?.getEntities() as EntityType[], - } - return genericRequest; - } - - private async decodeBase64AndSaveToFile(base64Data: string, outputFilePath: string) { - try { - const buffer = Buffer.from(base64Data, ENCODING_TYPE.BASE64); - await fs.promises.writeFile(outputFilePath, buffer); - } catch (error) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); - } - } - - private async processDeidentifyFileResponse(response: DeidentifyFileDetectRunResponse, outputDirectory: string, fileBaseName: string) { - await fs.promises.mkdir(outputDirectory, { recursive: true }); - - for (const fileObject of response.output) { - const { processedFile, processedFileExtension } = fileObject as DeidentifyFileOutput; - - if (!processedFile || !processedFileExtension) { - continue; - } - - if (!/^[a-zA-Z0-9]+$/.test(processedFileExtension)) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); - } - - const outputFileName = `processed-${fileBaseName}.${processedFileExtension}`; - const outputFilePath = path.join(outputDirectory, outputFileName); - const resolvedOutput = path.resolve(outputFilePath); - const resolvedDir = path.resolve(outputDirectory); - if (!resolvedOutput.startsWith(resolvedDir + path.sep) && resolvedOutput !== resolvedDir) { - throw new SkyflowError(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); - } - - if (processedFileExtension === FILE_EXTENSION.JSON) { - const jsonData = Buffer.from(processedFile, ENCODING_TYPE.BASE64).toString(ENCODING_TYPE.UTF8); - await fs.promises.writeFile(outputFilePath, jsonData); - } else if (processedFileExtension === FILE_EXTENSION.MP3 || processedFileExtension === FILE_EXTENSION.WAV) { - const mp3Data = Buffer.from(processedFile, ENCODING_TYPE.BASE64); - await fs.promises.writeFile(outputFilePath, mp3Data, { encoding: ENCODING_TYPE.BINARY }); - } else { - await this.decodeBase64AndSaveToFile(processedFile, outputFilePath); - } - } - } - - private getReqType(format: string): DeidenitfyFileRequestTypes{ - let reqType: DeidenitfyFileRequestTypes - if (Object.values(DeidentifyAudioRequestFileDataFormat).includes(format as DeidentifyAudioRequestFileDataFormat)){ - reqType = DeidenitfyFileRequestTypes.AUDIO; - } else if (format.includes(DeidenitfyFileRequestTypes.PDF.toLowerCase())){ - reqType = DeidenitfyFileRequestTypes.PDF - } else if (format.includes(DeidenitfyFileRequestTypes.TEXT.toLowerCase())){ - reqType = DeidenitfyFileRequestTypes.TEXT - } else if (Object.values(DeidentifyImageRequestFileDataFormat).includes(format as DeidentifyImageRequestFileDataFormat)){ - reqType = DeidenitfyFileRequestTypes.IMAGE; - } else if (Object.values(DeidentifyPresentationRequestFileDataFormat).includes(format as DeidentifyPresentationRequestFileDataFormat)){ - reqType = DeidenitfyFileRequestTypes.PPT; - } else if (Object.values(DeidentifySpreadsheetRequestFileDataFormat).includes(format as DeidentifySpreadsheetRequestFileDataFormat)){ - reqType = DeidenitfyFileRequestTypes.SPREADSHEET; - } else if (Object.values(DeidentifyStructuredTextRequestFileDataFormat).includes(format as DeidentifyStructuredTextRequestFileDataFormat)){ - reqType = DeidenitfyFileRequestTypes.STRUCTURED_TEXT; - } else if (Object.values(DeidentifyDocumentRequestFileDataFormat).includes(format as DeidentifyDocumentRequestFileDataFormat)){ - reqType = DeidenitfyFileRequestTypes.DOCUMENT; - } else if (Object.values(DeidentifyFileRequestFileDataFormat).includes(format as DeidentifyFileRequestFileDataFormat)){ - reqType = DeidenitfyFileRequestTypes.FILE - } else { - reqType = DeidenitfyFileRequestTypes.FILE - } - return reqType; - } - - private pollForProcessedFile(runId: string, req: GetRunRequest, maxWaitTime: number, resolve: Function, reject: Function): void { - let currentWaitTime = 1; // Start with 1 second - - const poll = () => { - this.client.filesAPI.getRun(runId, req) - .then((response: DeidentifyStatusResponse) => { - if (response.status?.toUpperCase() === DETECT_STATUS.IN_PROGRESS -) { - if (currentWaitTime >= maxWaitTime) { - resolve({ data: { status: 'IN_PROGRESS' }, runId }); - resolve({ data: { status: 'IN_PROGRESS' }, runId }); - } else { - const nextWaitTime = currentWaitTime * 2; - let waitTime = 0; - if (nextWaitTime >= maxWaitTime) { - waitTime = maxWaitTime - currentWaitTime; - currentWaitTime = maxWaitTime; - } else { - waitTime = nextWaitTime; - currentWaitTime = nextWaitTime; - } - setTimeout(() => { - poll(); - }, waitTime * 1000); - } - } else if (response.status?.toUpperCase() === DETECT_STATUS.SUCCESS) { - resolve({ data: response, runId }); - } - else if (response.status?.toUpperCase() === DETECT_STATUS.FAILED) { - reject(new SkyflowError(SKYFLOW_ERROR_CODE.INTERNAL_SERVER_ERROR, [response.message])); - } else { - reject(new SkyflowError(SKYFLOW_ERROR_CODE.INTERNAL_SERVER_ERROR, [response.message])); - } - }) - .catch((error) => { - reject(error); - }); - }; - - poll(); // Start polling - } - private handleRequest(apiCall: Function, requestType: string, waitTime: number = 64): Promise { - return new Promise((resolve, reject) => { - printLog(parameterizedString(logs.infoLogs.EMIT_REQUEST, TYPES[requestType]), MessageType.LOG, this.client.getLogLevel()); - const sdkHeaders = this.createSdkHeaders(); - - getBearerToken(this.client.getCredentials(), this.client.getLogLevel()).then(authInfo => { - this.client.initAPI(authInfo, requestType); - apiCall({ headers: { ...sdkHeaders } }) - .then((response: any) => { - const { data, rawResponse } = response; - const requestId = rawResponse?.headers?.get('x-request-id'); - printLog(logs.infoLogs[`${requestType}_REQUEST_RESOLVED`], MessageType.LOG, this.client.getLogLevel()); - switch (requestType) { - case TYPES.DEIDENTIFY_TEXT: - case TYPES.REIDENTIFY_TEXT: - resolve({records: data, requestId} as T) - break; - case TYPES.DEIDENTIFY_FILE: - const req: GetRunRequest = { - vault_id: this.client.vaultId, - } - - const maxWaitTime = waitTime; - - this.pollForProcessedFile(data?.run_id, req, maxWaitTime, resolve, reject); // Call the extracted polling function - break; - case TYPES.DETECT_RUN: - resolve({data, requestId} as T) - break; - - } - }).catch((error: SkyflowAllError) => { - printLog(logs.errorLogs[`${requestType}_REQUEST_REJECTED`], MessageType.ERROR, this.client.getLogLevel()); - this.client.failureResponse(error).catch((err) => reject(err)) - }); - }).catch(reject); - }); - } - - private buildDeidentifyTextRequest(request: DeidentifyTextRequest, options?: DeidentifyTextOptions) { - return { - vault_id: this.client.vaultId, - text: request.text, - entity_types: options?.getEntities() as EntityType[], - allow_regex: options?.getAllowRegexList(), - restrict_regex: options?.getRestrictRegexList(), - token_type: { - default: options?.getTokenFormat()?.getDefault(), - vault_token: options?.getTokenFormat()?.getVaultToken(), - entity_unq_counter: options?.getTokenFormat()?.getEntityUniqueCounter(), - entity_only: options?.getTokenFormat()?.getEntityOnly(), - } as TokenType, - transformations: this.getTransformations(options) as GeneratedTransformations, - }; - } - - private parseDeidentifyTextResponse(records: DeidentifyStringResponse) { - return { - processedText: records?.processed_text, - entities: (records?.entities ?? []).map((entity: DetectedEntity) => ({ - token: entity?.token, - value: entity?.value, - textIndex: { - start: entity?.location?.start_index, - end: entity?.location?.end_index, - }, - processedIndex: { - start: entity?.location?.start_index_processed, - end: entity?.location?.end_index_processed, - }, - entity: entity?.entity_type, - scores: entity?.entity_scores, - })), - wordCount: records?.word_count, - charCount: records?.character_count, - errors: null, - }; - } - - private parseDeidentifyFileResponse(data: DeidentifyFileDetectRunResponse, runId?: string, status?: string): DeidentifyFileResponse { - const base64String = data.output?.[0]?.processedFile ?? ''; - const extension = data.output?.[0]?.processedFileExtension ?? ''; - - let file: File | undefined = undefined; - - if (base64String && extension) { - const buffer = Buffer.from(base64String, ENCODING_TYPE.BASE64); - const fileName = `deidentified.${extension}`; - file = new File([buffer], fileName); -} - - return new DeidentifyFileResponse({ - fileBase64: base64String, - file: file, - type: data.output?.[0]?.processedFileType ?? data.outputType ?? "", - extension: extension, - wordCount: data.wordCharacterCount?.wordCount ?? 0, - charCount: data.wordCharacterCount?.characterCount ?? 0, - sizeInKb: data.size ?? 0, - durationInSeconds: data.duration ?? 0, - pageCount: data.pages ?? 0, - slideCount: data.slides ?? 0, - entities: (data.output || []) - .filter((fileObject: DeidentifyFileOutput) => fileObject.processedFileType === FILE_PROCESSING.ENTITIES) - .map((fileObject: DeidentifyFileOutput) => ({ - file: fileObject.processedFile as string, - extension: fileObject.processedFileExtension as string, - })), - runId: data.runId ?? runId, - status: status, - }); - } - - deidentifyText(request: DeidentifyTextRequest, options?: DeidentifyTextOptions): Promise { - return new Promise((resolve, reject) => { - try { - printLog(logs.infoLogs.DEIDENTIFY_TEXT_TRIGGERED, MessageType.LOG, this.client.getLogLevel()); - printLog(logs.infoLogs.VALIDATE_DEIDENTIFY_TEXT_INPUT, MessageType.LOG, this.client.getLogLevel()); - - validateDeIdentifyTextRequest(request, options, this.client.getLogLevel()); - - const requestBody = this.buildDeidentifyTextRequest(request, options); - this.handleRequest>( - () => this.client.stringsAPI.deidentifyString( - requestBody - ).withRawResponse(), - TYPES.DEIDENTIFY_TEXT - ).then(data => { - const parsedResponse = new DeidentifyTextResponse(this.parseDeidentifyTextResponse(data.records)) - resolve(parsedResponse); - }).catch(error => { - reject(error) - }); - } catch (error) { - if (error instanceof Error) - printLog(removeSDKVersion(error.message), MessageType.ERROR, this.client.getLogLevel()); - reject(error); - } - }); - } - - reidentifyText(request: ReidentifyTextRequest, options?: ReidentifyTextOptions): Promise { - return new Promise((resolve, reject) => { - try { - printLog(logs.infoLogs.REIDENTIFY_TEXT_TRIGGERED, MessageType.LOG, this.client.getLogLevel()); - printLog(logs.infoLogs.VALIDATE_REIDENTIFY_TEXT_INPUT, MessageType.LOG, this.client.getLogLevel()); - validateReidentifyTextRequest(request, options, this.client.getLogLevel()); - - const requestBody = { - text: request.text, - vault_id: this.client.vaultId, - format: { - redacted: options?.getRedactedEntities(), - masked: options?.getMaskedEntities(), - plaintext: options?.getPlainTextEntities(), - } - }; - this.handleRequest>>( - () => this.client.stringsAPI.reidentifyString( - requestBody - ).withRawResponse(), - TYPES.REIDENTIFY_TEXT - ).then(data => { - resolve(new ReidentifyTextResponse({ - processedText: data.records.text - })); - }).catch(error => { - reject(error) - }); - - } catch (error) { - if (error instanceof Error) - printLog(removeSDKVersion(error.message), MessageType.ERROR, this.client.getLogLevel()); - reject(error); - } - }); + super(client); } - getDetectRun(request: GetDetectRunRequest): Promise { - return new Promise((resolve, reject) => { - try { - printLog(logs.infoLogs.GET_DETECT_RUN_TRIGGERED, MessageType.LOG, this.client.getLogLevel()); - printLog(logs.infoLogs.VALIDATE_GET_DETECT_RUN_INPUT, MessageType.LOG, this.client.getLogLevel()); - validateGetDetectRunRequest(request, this.client.getLogLevel()); - - const req: GetRunRequest = { - vault_id: this.client.vaultId - } - - this.handleRequest>( - () => this.client.filesAPI.getRun( - request.runId, - req - ).withRawResponse(), - TYPES.DETECT_RUN - ).then(response => { - const deidentifiedFileResponse = this.parseDeidentifyFileResponse(response.data, request.runId, response.data.status); - resolve(deidentifiedFileResponse); - }).catch(error => { - reject(error) - }); - } catch (error) { - if (error instanceof Error) - printLog(removeSDKVersion(error.message), MessageType.ERROR, this.client.getLogLevel()); - reject(error); - } - }); + protected toDeidentifySource(value: string | null | undefined): DeidentifySourceType | null { + return BaseDetectController.enumOrNull(DeidentifySourceType, value); } - async deidentifyFile(request: DeidentifyFileRequest, options?: DeidentifyFileOptions): Promise { - try { - printLog(logs.infoLogs.DETECT_FILE_TRIGGERED, MessageType.LOG, this.client.getLogLevel()); - printLog(logs.infoLogs.VALIDATE_DETECT_FILE_INPUT, MessageType.LOG, this.client.getLogLevel()); - validateDeidentifyFileRequest(request, options, this.client.getLogLevel()); - - const fileObj = await this.getFileFromRequest(request); - - const fileName = fileObj.name; - const fileBaseName = path.parse(fileName).name; - const fileExtension = fileName.substring(fileName.lastIndexOf('.') + 1); - - const waitTime = options?.getWaitTime() ?? 64; - - const reqType : DeidenitfyFileRequestTypes = this.getReqType(fileExtension); - type PollResult = - | { data: DeidentifyFileDetectRunResponse; runId: string } - | { data: { status: string }; runId: string }; - let promiseReq: Promise; - switch (reqType){ - case DeidenitfyFileRequestTypes.AUDIO: - promiseReq = this.buildAudioRequest(fileObj, options, fileExtension) - .then((audioReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyAudio( - audioReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - case DeidenitfyFileRequestTypes.TEXT: - promiseReq = this.buildTextFileRequest(fileObj, options) - .then((textFileReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyText( - textFileReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - case DeidenitfyFileRequestTypes.PDF: - promiseReq = this.buildPdfRequest(fileObj, options) - .then((pdfReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyPdf( - pdfReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - case DeidenitfyFileRequestTypes.IMAGE: - promiseReq = this.buildImageRequest(fileObj, options, fileExtension) - .then((imageReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyImage( - imageReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - case DeidenitfyFileRequestTypes.PPT: - promiseReq = this.buildPptRequest(fileObj, options, fileExtension) - .then((pptReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyPresentation( - pptReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - case DeidenitfyFileRequestTypes.SPREADSHEET: - promiseReq = this.buildSpreadsheetRequest(fileObj, options, fileExtension) - .then((spreadsheetReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifySpreadsheet( - spreadsheetReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - case DeidenitfyFileRequestTypes.STRUCTURED_TEXT: - promiseReq = this.buildStructuredTextRequest(fileObj, options, fileExtension) - .then((structuredTextReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyStructuredText( - structuredTextReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - case DeidenitfyFileRequestTypes.DOCUMENT: - promiseReq = this.buildDocumentRequest(fileObj, options, fileExtension) - .then((documentReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyDocument( - documentReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - default: - promiseReq = this.buildGenericFileRequest(fileObj, options, fileExtension) - .then((defaultReq) => { - return this.handleRequest( - () => this.client.filesAPI.deidentifyFile( - defaultReq - ).withRawResponse(), - TYPES.DEIDENTIFY_FILE, - waitTime - ); - }); - break; - } - - const { data, runId } = await promiseReq; - if(runId && data.status === DETECT_STATUS.IN_PROGRESS) { - return new DeidentifyFileResponse({ - runId: runId, - status: data.status, - }); - } - const fullResponse = data as DeidentifyFileDetectRunResponse; - if (options?.getOutputDirectory() && fullResponse.status === DETECT_STATUS.SUCCESS) { - await this.processDeidentifyFileResponse(fullResponse, options.getOutputDirectory() as string, fileBaseName); - } - const deidentifiedFileResponse = this.parseDeidentifyFileResponse(fullResponse, runId, fullResponse.status); - return deidentifiedFileResponse; - } catch (error) { - if (error instanceof Error) - printLog(removeSDKVersion(error.message), MessageType.ERROR, this.client.getLogLevel()); - throw error; - } + protected toReidentifySource(value: string | null | undefined): ReidentifySourceType | null { + return BaseDetectController.enumOrNull(ReidentifySourceType, value); } } diff --git a/skyvault/src/vault/detect/types.ts b/skyvault/src/vault/detect/types.ts new file mode 100644 index 00000000..03ed5cde --- /dev/null +++ b/skyvault/src/vault/detect/types.ts @@ -0,0 +1,39 @@ +// Detect V2 types whose values depend on the vault type. These are the skyvault versions; the +// flowvault SDK ships its own. The shared request and response models in skyflow-common are typed +// against the contracts `DeidentifyFileSource`, `ReidentifyFileSource` and `RedactionRule`, which +// these satisfy. +import type { EntityType, RedactionRule, RedactionType } from "skyflow-common"; + +/** + * How the input file of `deidentifyFile` is supplied, and how a run delivers its outputs. + * Skyvault vaults accept a base64 string, the skyflow id of a file already stored in the vault, + * or a presigned URL. A local `file` or `filePath` is always sent as `BASE64`. + */ +export enum DeidentifySourceType { + BASE64 = "BASE64", + SKYFLOW_ID = "SKYFLOW_ID", + PRESIGNED_URL = "PRESIGNED_URL", +} + +/** + * How the input file of `reidentifyFile` is supplied, and how its outputs are delivered. + * Re-identification does not accept a skyflow id as the source. + */ +export enum ReidentifySourceType { + BASE64 = "BASE64", + PRESIGNED_URL = "PRESIGNED_URL", +} + +/** + * One redaction rule for `reidentifyString` and `reidentifyFile` on a skyvault vault: the entity + * type the rule applies to and how its value is rendered. Both fields are optional on the API; + * the server rejects combinations it does not support. + */ +export interface RedactionLevel extends RedactionRule { + /** Entity type the rule applies to. */ + entityName?: EntityType; + /** How the value is rendered. */ + redactionType?: RedactionType; + tokenGroupName?: never; + redactionPattern?: never; +} diff --git a/skyvault/src/vault/model/options/deidentify-file/bleep-audio/index.ts b/skyvault/src/vault/model/options/deidentify-file/bleep-audio/index.ts deleted file mode 100644 index a50326f2..00000000 --- a/skyvault/src/vault/model/options/deidentify-file/bleep-audio/index.ts +++ /dev/null @@ -1,31 +0,0 @@ -export class Bleep { - private _gain?: number; - private _frequency?: number; - private _startPadding?: number; - private _stopPadding?: number; - - getGain(): number | undefined { - return this._gain; - } - setGain(gain: number) { - this._gain = gain; - } - getFrequency(): number | undefined { - return this._frequency; - } - setFrequency(frequency: number) { - this._frequency = frequency; - } - getStartPadding(): number | undefined { - return this._startPadding; - } - setStartPadding(startPadding: number) { - this._startPadding = startPadding; - } - getStopPadding(): number | undefined { - return this._stopPadding; - } - setStopPadding(stopPadding: number) { - this._stopPadding = stopPadding; - } -} \ No newline at end of file diff --git a/skyvault/src/vault/model/options/deidentify-file/index.ts b/skyvault/src/vault/model/options/deidentify-file/index.ts deleted file mode 100644 index 25c0ecc3..00000000 --- a/skyvault/src/vault/model/options/deidentify-file/index.ts +++ /dev/null @@ -1,147 +0,0 @@ -import { DetectEntities, DetectOutputTranscription, MaskingMethod } from "skyflow-common"; -import TokenFormat from "../deidentify-text/token-format"; -import Transformations from "../deidentify-text/transformations"; -import { Bleep } from './bleep-audio' - - -class DeidentifyFileOptions { - private _entities?: DetectEntities[]; - private _allowRegexList?: string[]; - private _restrictRegexList?: string[]; - private _tokenFormat?: TokenFormat; - private _transformations?: Transformations; - private _outputProcessedImage?: boolean; - private _outputOcrText?: boolean; - private _maskingMethod?: MaskingMethod; - private _pixelDensity?: number; - private _maxResolution?: number; - private _outputProcessedAudio?: boolean; - private _outputTranscription?: DetectOutputTranscription; - private _bleep?: Bleep; - private _outputDirectory?: string; - private _waitTime?: number; - - constructor() { - } - - getEntities(): string[] | undefined { - return this._entities; - } - - // Setters - setEntities(entities: DetectEntities[]) { - this._entities = entities; - } - - getAllowRegexList(): string[] | undefined { - return this._allowRegexList; - } - - setAllowRegexList(allowRegexList: string[]) { - this._allowRegexList = allowRegexList; - } - - getRestrictRegexList(): string[] | undefined{ - return this._restrictRegexList; - } - - setRestrictRegexList(restrictRegexList: string[]) { - this._restrictRegexList = restrictRegexList; - } - - getTokenFormat(): TokenFormat | undefined{ - return this._tokenFormat; - } - - setTokenFormat(tokenFormat: TokenFormat) { - this._tokenFormat = tokenFormat; - } - - getTransformations(): Transformations | undefined{ - return this._transformations; - } - - setTransformations(transformations: Transformations) { - this._transformations = transformations; - } - - getOutputProcessedImage(): boolean | undefined { - return this._outputProcessedImage; - } - - setOutputProcessedImage(value: boolean | undefined) { - this._outputProcessedImage = value; - } - - getOutputOcrText(): boolean | undefined { - return this._outputOcrText; - } - - setOutputOcrText(outputOcrText: boolean | undefined) { - this._outputOcrText = outputOcrText; - } - - getMaskingMethod(): MaskingMethod | undefined { - return this._maskingMethod; - } - - setMaskingMethod(maskingMethod: MaskingMethod | undefined) { - this._maskingMethod = maskingMethod; - } - - getPixelDensity(): number | undefined { - return this._pixelDensity; - } - - setPixelDensity(pixelDensity: number | undefined) { - this._pixelDensity = pixelDensity; - } - - getMaxResolution(): number | undefined { - return this._maxResolution; - } - - setMaxResolution(maxResolution: number | undefined) { - this._maxResolution = maxResolution; - } - - getOutputProcessedAudio(): boolean | undefined { - return this._outputProcessedAudio; - } - - setOutputProcessedAudio(outputProcessedAudio: boolean | undefined) { - this._outputProcessedAudio = outputProcessedAudio; - } - - getOutputTranscription(): DetectOutputTranscription | undefined { - return this._outputTranscription; - } - - setOutputTranscription(outputTranscription: DetectOutputTranscription | undefined) { - this._outputTranscription = outputTranscription; - } - - getBleep(): Bleep | undefined { - return this._bleep; - } - - setBleep(bleep: Bleep | undefined) { - this._bleep = bleep; - } - getOutputDirectory(): string | undefined { - return this._outputDirectory; - } - setOutputDirectory(outputDirectory: string | undefined) { - this._outputDirectory = outputDirectory; - } - - getWaitTime(): number | undefined { - return this._waitTime; - } - - setWaitTime(waitTime: number | undefined) { - this._waitTime = waitTime; - } -} - -export default DeidentifyFileOptions; diff --git a/skyvault/src/vault/model/options/deidentify-text/index.ts b/skyvault/src/vault/model/options/deidentify-text/index.ts deleted file mode 100644 index 10e446e0..00000000 --- a/skyvault/src/vault/model/options/deidentify-text/index.ts +++ /dev/null @@ -1,56 +0,0 @@ -import { DetectEntities } from "skyflow-common"; -import TokenFormat from "./token-format"; -import Transformations from "./transformations"; - -class DeidentifyTextOptions { - // Fields - private _entities?: DetectEntities[]; - private _allowRegexList?: string[]; - private _restrictRegexList?: string[]; - private _tokenFormat?: TokenFormat; - private _transformations?: Transformations; - - // Setters - setEntities(entities: DetectEntities[]) { - this._entities = entities; - } - - setAllowRegexList(allowRegexList: string[]) { - this._allowRegexList = allowRegexList; - } - - setRestrictRegexList(restrictRegexList: string[]) { - this._restrictRegexList = restrictRegexList; - } - - setTokenFormat(tokenFormat: TokenFormat) { - this._tokenFormat = tokenFormat; - } - - setTransformations(transformations: Transformations) { - this._transformations = transformations; - } - - // Getters - getEntities(): DetectEntities[] | undefined { - return this._entities; - } - - getAllowRegexList(): string[] | undefined { - return this._allowRegexList; - } - - getRestrictRegexList(): string[] | undefined { - return this._restrictRegexList; - } - - getTokenFormat(): TokenFormat | undefined { - return this._tokenFormat; - } - - getTransformations(): Transformations | undefined { - return this._transformations; - } -} - -export default DeidentifyTextOptions; \ No newline at end of file diff --git a/skyvault/src/vault/model/options/deidentify-text/token-format/index.ts b/skyvault/src/vault/model/options/deidentify-text/token-format/index.ts deleted file mode 100644 index 8ee2e7e1..00000000 --- a/skyvault/src/vault/model/options/deidentify-text/token-format/index.ts +++ /dev/null @@ -1,45 +0,0 @@ -import { DetectEntities, TokenType } from "skyflow-common"; - -class TokenFormat { - // Fields - private _default?: TokenType = TokenType.ENTITY_UNIQUE_COUNTER; - private _vaultToken?: DetectEntities[]; - private _entityUniqueCounter?: DetectEntities[]; - private _entityOnly?: DetectEntities[]; - - // Setters - setDefault(defaultToken: TokenType) { - this._default = defaultToken; - } - - setVaultToken(vaultToken: DetectEntities[]) { - this._vaultToken = vaultToken; - } - - setEntityUniqueCounter(entityUniqueCounter: DetectEntities[]) { - this._entityUniqueCounter = entityUniqueCounter; - } - - setEntityOnly(entityOnly: DetectEntities[]) { - this._entityOnly = entityOnly; - } - - // Getters - getDefault(): string | undefined { - return this._default; - } - - getVaultToken(): string[] | undefined { - return this._vaultToken; - } - - getEntityUniqueCounter(): string[] | undefined { - return this._entityUniqueCounter; - } - - getEntityOnly(): string[] | undefined { - return this._entityOnly; - } -} - -export default TokenFormat; \ No newline at end of file diff --git a/skyvault/src/vault/model/options/deidentify-text/transformations/index.ts b/skyvault/src/vault/model/options/deidentify-text/transformations/index.ts deleted file mode 100644 index d589c074..00000000 --- a/skyvault/src/vault/model/options/deidentify-text/transformations/index.ts +++ /dev/null @@ -1,22 +0,0 @@ -import { DetectEntities } from "skyflow-common"; - -class Transformations { - // Fields - private _shiftDays?: { - max: number; - min: number; - entities: DetectEntities[]; - }; - - // Setters - setShiftDays(shiftDays: { max: number; min: number; entities: DetectEntities[] }) { - this._shiftDays = shiftDays; - } - - // Getters - getShiftDays(): { max: number; min: number; entities: DetectEntities[] } | undefined { - return this._shiftDays; - } -} - -export default Transformations; \ No newline at end of file diff --git a/skyvault/src/vault/model/options/reidentify-text/index.ts b/skyvault/src/vault/model/options/reidentify-text/index.ts deleted file mode 100644 index 78659816..00000000 --- a/skyvault/src/vault/model/options/reidentify-text/index.ts +++ /dev/null @@ -1,36 +0,0 @@ -import { DetectEntities } from "skyflow-common"; - -class ReidentifyTextOptions { - // Fields - private _redactedEntities?: DetectEntities[]; - private _maskedEntities?: DetectEntities[]; - private _plainTextEntities?: DetectEntities[]; - - // Setters - setRedactedEntities(redactedEntities: DetectEntities[]) { - this._redactedEntities = redactedEntities; - } - - setMaskedEntities(maskedEntities: DetectEntities[]) { - this._maskedEntities = maskedEntities; - } - - setPlainTextEntities(plainTextEntities: DetectEntities[]) { - this._plainTextEntities = plainTextEntities; - } - - // Getters - getRedactedEntities(): DetectEntities[] | undefined { - return this._redactedEntities; - } - - getMaskedEntities(): DetectEntities[] | undefined { - return this._maskedEntities; - } - - getPlainTextEntities(): DetectEntities[] | undefined { - return this._plainTextEntities; - } -} - -export default ReidentifyTextOptions; \ No newline at end of file diff --git a/skyvault/src/vault/model/request/deidentify-file/index.ts b/skyvault/src/vault/model/request/deidentify-file/index.ts deleted file mode 100644 index 272e89a2..00000000 --- a/skyvault/src/vault/model/request/deidentify-file/index.ts +++ /dev/null @@ -1,19 +0,0 @@ -import { FileInput } from "../../../types"; - -class DeidentifyFileRequest { - private _file: FileInput ; // Accepts a native file object (File for browser, Buffer for Node.js) - - constructor(file: FileInput ) { - this._file = file; - } - - public getFile(): FileInput { - return this._file; - } - - public setFile(file: FileInput ): void { - this._file = file; - } -} - -export default DeidentifyFileRequest; \ No newline at end of file diff --git a/skyvault/src/vault/model/request/deidentify-text/index.ts b/skyvault/src/vault/model/request/deidentify-text/index.ts deleted file mode 100644 index 668a814c..00000000 --- a/skyvault/src/vault/model/request/deidentify-text/index.ts +++ /dev/null @@ -1,21 +0,0 @@ -class DeidentifyTextRequest { - // Field - private _text: string; - - // Constructor - constructor(text: string) { - this._text = text; - } - - // Getter for text - public get text(): string { - return this._text; - } - - // Setter for text - public set text(value: string) { - this._text = value; - } -} - -export default DeidentifyTextRequest; \ No newline at end of file diff --git a/skyvault/src/vault/model/request/get-detect-run/index.ts b/skyvault/src/vault/model/request/get-detect-run/index.ts deleted file mode 100644 index 563dabb8..00000000 --- a/skyvault/src/vault/model/request/get-detect-run/index.ts +++ /dev/null @@ -1,21 +0,0 @@ -class GetDetectRunRequest { - // Field - private _runId: string; - - // Constructor - constructor(runId: string) { - this._runId = runId; - } - - // Getter for runId - public get runId(): string { - return this._runId; - } - - // Setter for runId - public set runId(value: string) { - this._runId = value; - } -} - -export default GetDetectRunRequest; \ No newline at end of file diff --git a/skyvault/src/vault/model/request/reidentify-text/index.ts b/skyvault/src/vault/model/request/reidentify-text/index.ts deleted file mode 100644 index d83d3809..00000000 --- a/skyvault/src/vault/model/request/reidentify-text/index.ts +++ /dev/null @@ -1,21 +0,0 @@ -class ReidentifyTextRequest { - // Field - private _text: string; - - // Constructor - constructor(text: string) { - this._text = text; - } - - // Getter for text - public get text(): string { - return this._text; - } - - // Setter for text - public set text(value: string) { - this._text = value; - } -} - -export default ReidentifyTextRequest; \ No newline at end of file diff --git a/skyvault/src/vault/model/response/deidentify-file/index.ts b/skyvault/src/vault/model/response/deidentify-file/index.ts deleted file mode 100644 index 9732c40a..00000000 --- a/skyvault/src/vault/model/response/deidentify-file/index.ts +++ /dev/null @@ -1,75 +0,0 @@ -import { SkyflowRecordError } from "skyflow-common"; - -class DeidentifyFileResponse { - // fields - fileBase64?: string; - entities?: Array<{ - file: string; - extension: string; - }> = []; - file?: File; - type?: string; - extension?: string; - wordCount?: number; - charCount?: number; - sizeInKb?: number; - durationInSeconds?: number; - pageCount?: number; - slideCount?: number; - runId?: string; - status?: string; - errors: Array | null; - - constructor({ - fileBase64, - file, - type, - extension, - wordCount, - charCount, - sizeInKb, - durationInSeconds, - pageCount, - slideCount, - entities, - runId, - status, - errors, - } :{ - fileBase64?: string; - file?: File; - type?: string; - extension?: string; - wordCount?: number; - charCount?: number; - sizeInKb?: number; - durationInSeconds?: number; - pageCount?: number; - slideCount?: number; - entities?: Array<{ - file: string; - extension: string; - }>; - runId?: string; - status?: string; - errors?: Array | null; - }) { - this.fileBase64 = fileBase64; - this.file = file; - this.type = type; - this.extension = extension; - this.wordCount = wordCount; - this.charCount = charCount; - this.sizeInKb = sizeInKb; - this.durationInSeconds = durationInSeconds; - this.pageCount = pageCount; - this.slideCount = slideCount; - this.entities = entities; - this.runId = runId; - this.status = status; - this.errors = errors ?? null; - } - -} - -export default DeidentifyFileResponse; \ No newline at end of file diff --git a/skyvault/src/vault/model/response/deidentify-text/index.ts b/skyvault/src/vault/model/response/deidentify-text/index.ts deleted file mode 100644 index fb78984f..00000000 --- a/skyvault/src/vault/model/response/deidentify-text/index.ts +++ /dev/null @@ -1,51 +0,0 @@ -//imports - -import { IndexRange } from "../../../types"; -import { SkyflowRecordError } from "skyflow-common"; - -class DeidentifyTextResponse { - //fields - processedText: string; - entities: Array<{ - token?: string; - value?: string; - textIndex?: IndexRange; - processedIndex?: IndexRange; - entity?: string; - scores?: Record; - }>; - wordCount: number; - charCount: number; - errors: Array | null; - - constructor({ - processedText, - entities, - wordCount, - charCount, - errors, - }: { - processedText: string; - entities: Array<{ - token?: string; - value?: string; - textIndex?: IndexRange; - processedIndex?: IndexRange; - entity?: string; - scores?: Record; - }>; - wordCount: number; - charCount: number; - errors?: Array | null; - }) { - this.processedText = processedText; - this.entities = entities; - this.wordCount = wordCount; - this.charCount = charCount; - this.errors = errors ?? null; - } - - //getters and setters -} - -export default DeidentifyTextResponse; \ No newline at end of file diff --git a/skyvault/src/vault/model/response/reidentify-text/index.ts b/skyvault/src/vault/model/response/reidentify-text/index.ts deleted file mode 100644 index 0aea2b99..00000000 --- a/skyvault/src/vault/model/response/reidentify-text/index.ts +++ /dev/null @@ -1,16 +0,0 @@ -// imports - -class ReidentifyTextResponse { - //fields - processedText: string; - - constructor({ - processedText, - }: { - processedText: string; - }) { - this.processedText = processedText; - } -} - -export default ReidentifyTextResponse; \ No newline at end of file diff --git a/skyvault/src/vault/types/index.ts b/skyvault/src/vault/types/index.ts index 5e78002e..4d5ae19d 100644 --- a/skyvault/src/vault/types/index.ts +++ b/skyvault/src/vault/types/index.ts @@ -12,15 +12,11 @@ import { InsertResponseType } from "skyflow-common"; // public API surface of skyflow-node continues to expose them from here. export type { SkyflowConfig, - FileInput, - Filepath, - FileObject, InsertResponseType, GetResponseData, QueryResponseType, StringKeyValueMapType, TokenizeRequestType, - IndexRange, ServiceAccountResponseError, } from "skyflow-common"; @@ -121,15 +117,6 @@ export interface RecordsResponse> { requestId: string; } -export interface DetectTextResponse { - records: T; - requestId: string; -} - -export interface DetectFileResponse { - data: T; - requestId: string; -} export interface SkyflowIdResponse { skyflowId: string; /** @deprecated Renamed to skyflowId. Will be removed in v3. */ @@ -140,39 +127,3 @@ export interface TokensResponse extends SkyflowIdResponse { tokens?: Record; } -export type DeidentifyFileOutputProcessedFileType = - | "entities" - | "plaintext_transcription" - | "redacted_audio" - | "redacted_diarized_transcription" - | "redacted_file" - | "redacted_image" - | "redacted_medical_diarized_transcription" - | "redacted_medical_transcription" - | "redacted_text" - | "redacted_transcription"; - -export interface DeidentifyFileOutput { - processedFile?: string; - processedFileType?: DeidentifyFileOutputProcessedFileType; - processedFileExtension?: string; -} - -export type DeidentifyStatusResponseOutputType = "BASE64" | "UNKNOWN"; -export type WordCharacterCount = { - wordCount?: number; - characterCount?: number; -} - -export interface DeidentifyFileDetectRunResponse { - status: "FAILED" | "IN_PROGRESS" | "SUCCESS" | "UNKNOWN"; - output: DeidentifyFileOutput[]; - outputType: DeidentifyStatusResponseOutputType; - message: string; - wordCharacterCount?: WordCharacterCount; - size?: number; - duration?: number; - pages?: number; - slides?: number; - runId?: string; -} diff --git a/skyvault/test/utils/validations.test.js b/skyvault/test/utils/validations.test.js index de25251d..1cc24002 100644 --- a/skyvault/test/utils/validations.test.js +++ b/skyvault/test/utils/validations.test.js @@ -1,362 +1,15 @@ const fs = require('fs'); const path = require('path'); -const { validateDeidentifyFileRequest, validateDeidentifyFileOptions, validateInsertRequest, validateUpdateRequest, validateGetRequest, validateDetokenizeRequest, validateTokenizeRequest, validateDeleteRequest, validateUploadFileRequest, validateQueryRequest, validateDeIdentifyTextRequest, validateReidentifyTextRequest, validateGetDetectRunRequest, validateInvokeConnectionRequest, validateUpdateOptions, validateGetColumnRequest, isRedactionType, isByot, isOrderBy, isMethod, validateInsertOptions, validateGetOptions, validateDetokenizeOptions } = require('../../src/utils/validations'); +const { validateInsertRequest, validateUpdateRequest, validateGetRequest, validateDetokenizeRequest, validateTokenizeRequest, validateDeleteRequest, validateUploadFileRequest, validateQueryRequest, validateInvokeConnectionRequest, validateUpdateOptions, validateGetColumnRequest, isRedactionType, isByot, isOrderBy, isMethod, validateInsertOptions, validateGetOptions, validateDetokenizeOptions } = require('../../src/utils/validations'); // Generic (contract-agnostic) validators now live in skyflow-common. const { validateSkyflowConfig, validateVaultConfig, validateUpdateVaultConfig, validateSkyflowCredentials, validateConnectionConfig, validateUpdateConnectionConfig, validateCredentialsWithId, isEnv, isLogLevel, isValidAPIKey } = require('skyflow-common'); -const DeidentifyFileRequest = require('../../src/vault/model/request/deidentify-file').default; -const DeidentifyFileOptions = require('../../src/vault/model/options/deidentify-file').default; // NOTE: On main, `require('../../src/error/codes')` returned the ES module // wrapper (the codes live on `.default`), so `SKYFLOW_ERROR_CODE.` was // `undefined` and `toThrow(undefined)` only asserted "throws something". We // require the whole skyflow-common module here to preserve that exact behavior. const SKYFLOW_ERROR_CODE = require('skyflow-common'); -const { default: TokenFormat } = require('../../src/vault/model/options/deidentify-text/token-format'); -const { DetectEntities, Env, TokenMode, OrderByEnum, MaskingMethod } = require('skyflow-common'); -const { Bleep } = require('../../src/vault/model/options/deidentify-file/bleep-audio'); +const { Env, TokenMode, OrderByEnum } = require('skyflow-common'); const { LogLevel } = require('skyflow-common'); -const { default: Transformations } = require('../../src/vault/model/options/deidentify-text/transformations'); - -describe('validateDeidentifyFileRequest', () => { - let mockFile; - let mockFilePath; - - beforeEach(() => { - // Create a mock File object - mockFile = new File(['test content'], 'test.txt', { type: 'text/plain' }); - mockFilePath = '/path/to/file.txt'; - jest.spyOn(fs, 'existsSync').mockImplementation(() => true); - jest.spyOn(fs, 'lstatSync').mockImplementation(() => ({ - isFile: () => true, - isDirectory: () => true - })); - }); - - afterEach(() => { - jest.restoreAllMocks(); - }); - - test('should throw error when request is null', () => { - expect(() => validateDeidentifyFileRequest(null)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); - }); - - test('should throw error when request is undefined', () => { - expect(() => validateDeidentifyFileRequest(undefined)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_REQUEST); - }); - - test('should throw error when neither file nor filePath is provided', () => { - const request = new DeidentifyFileRequest({}); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_INPUT); - }); - - test('should throw error when both file and filePath are provided', () => { - const request = new DeidentifyFileRequest({ - file: mockFile, - filePath: mockFilePath - }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_INPUT); - }); - - // File object validation tests - test('should validate request with valid file object', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - expect(() => validateDeidentifyFileRequest(request)).not.toThrow(); - }); - - test('should throw error for invalid file object', () => { - const request = new DeidentifyFileRequest({ - file: { invalid: 'file' } - }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - }); - - test('should throw error for file with empty name', () => { - const invalidFile = new File(['content'], '', { type: 'text/plain' }); - const request = new DeidentifyFileRequest({ file: invalidFile }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - }); - - test('should throw error when file is not a File instance', () => { - const request = new DeidentifyFileRequest({ - file: { name: 'test.txt', type: 'text/plain' } - }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - }); - - test('should throw error when file name is invalid', () => { - const invalidFile = new File(['content'], ' ', { type: 'text/plain' }); - const request = new DeidentifyFileRequest({ file: invalidFile }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - }); - - // File path validation tests - test('should validate request with valid file path', () => { - const request = new DeidentifyFileRequest({ filePath: mockFilePath }); - expect(() => validateDeidentifyFileRequest(request)).not.toThrow(); - }); - - test('should throw error for non-existent file path', () => { - jest.spyOn(fs, 'existsSync').mockImplementation(() => false); - const request = new DeidentifyFileRequest({ - filePath: '/invalid/path.txt' - }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_PATH); - }); - - test('should throw error when file path points to directory', () => { - jest.spyOn(fs, 'lstatSync').mockImplementation(() => ({ - isFile: () => false - })); - const request = new DeidentifyFileRequest({ - filePath: '/path/to/directory' - }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_PATH); - }); - - test('should throw error for empty file path', () => { - const request = new DeidentifyFileRequest({ filePath: '' }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_PATH); - }); - - // Options validation tests - test('should validate request with options containing valid wait time', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setWaitTime(30); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - test('should throw error for invalid wait time', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setWaitTime(65); // More than maximum allowed - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(); - }); - - test('should validate request with valid output directory', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/valid/output/dir'); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - test('should throw error when output directory does not exist', () => { - jest.spyOn(fs, 'existsSync').mockImplementation(() => false); - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/nonexistent/dir'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_DIRECTORY_PATH); - }); - - test('should validate request with valid entities array', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setEntities(['PERSON', 'LOCATION']); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - test('should throw error for invalid entities type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setEntities('PERSON'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_ENTITIES); - }); - - test('should validate request with valid allow regex list', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setAllowRegexList(['^test', 'pattern$']); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - test('should throw error for invalid allow regex list', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setAllowRegexList('pattern'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_ALLOW_REGEX_LIST); - }); - - test('should validate request with valid entities array', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setEntities(['PERSON', 'LOCATION']); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - test('should throw error for invalid entities type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setEntities('PERSON'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_ENTITIES); - }); - - test('should validate request with valid allow regex list', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setAllowRegexList(['^test', 'pattern$']); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - - - test('should throw error when options is invalid', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - expect(() => validateDeidentifyFileRequest(request, {})) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_OPTIONS); - }); - - test('should throw error for invalid transformations', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setTransformations({}); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_TRANSFORMATIONS); - }); - - test('should throw error for invalid output processed image type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setOutputProcessedImage('true'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_PROCESSED_IMAGE); - }); - - - - test('should throw error for invalid masking method type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setMaskingMethod(123); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_MASKING_METHOD); - }); - - test('should throw error for invalid pixel density type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setPixelDensity('300'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_PIXEL_DENSITY); - }); - - test('should throw error for invalid max resolution type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setMaxResolution('1024'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_MAX_RESOLUTION); - }); - - test('should throw error for invalid output processed audio type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setOutputProcessedAudio('true'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_PROCESSED_AUDIO); - }); - - test('should throw error for invalid output transcription type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setOutputTranscription(123); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_TRANSCRIPTION); - }); - - test("should throw error in case of vault token in deidentify file request", () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - const tokenFormat = new TokenFormat(); - tokenFormat.setVaultToken(DetectEntities.CREDIT_CARD); - options.setTokenFormat(tokenFormat); - - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_VAULT_TOKEN); - }); - - test('should validate request with all valid parameters', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setWaitTime(30); - options.setOutputDirectory('/valid/output/dir'); - options.setEntities(['PERSON', 'LOCATION']); - options.setAllowRegexList(['^test', 'pattern$']); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - test('should throw error when file base name is whitespace only', () => { - const invalidFile = new File(['content'], ' .txt', { type: 'text/plain' }); - const request = new DeidentifyFileRequest({ file: invalidFile }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_FILE_TYPE); - }); - - test('should throw error when file path is whitespace only', () => { - const request = new DeidentifyFileRequest({ filePath: ' ' }); - expect(() => validateDeidentifyFileRequest(request)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_PATH); - }); - - test('should throw error when output directory is not a string', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory(123); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_DIRECTORY); - }); - - test('should throw error for invalid restrict regex list in file options', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setRestrictRegexList('pattern'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_RESTRICT_REGEX_LIST); - }); - - test('should throw error when token format is not a TokenFormat instance in file options', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setTokenFormat({}); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_TOKEN_FORMAT); - }); - - test('should throw error for invalid output ocr text type', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setOutputOcrText('true'); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_OUTPUT_OCR_TEXT); - }); - - test('should throw error when bleep is not a Bleep instance', () => { - const request = new DeidentifyFileRequest({ file: mockFile }); - const options = new DeidentifyFileOptions(); - options.setBleep({}); - expect(() => validateDeidentifyFileRequest(request, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_BLEEP); - }); - - test('should throw error when validateDeidentifyFileOptions receives null', () => { - expect(() => validateDeidentifyFileOptions(null)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_DEIDENTIFY_FILE_OPTIONS); - }); -}); describe('validateSkyflowConfig', () => { // Test for null/undefined config @@ -2834,374 +2487,6 @@ describe('validateQueryRequest', () => { }); }); -describe('validateDeIdentifyTextRequest', () => { - // Test for invalid text - test('should throw error when text is missing', () => { - const request = {}; - expect(() => validateDeIdentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_DEIDENTIFY); - }); - - test('should throw error when text is empty', () => { - const request = { - text: '' - }; - expect(() => validateDeIdentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_DEIDENTIFY); - }); - - test('should throw error when text is whitespace', () => { - const request = { - text: ' ' - }; - expect(() => validateDeIdentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_DEIDENTIFY); - }); - - test('should throw error when text is not string', () => { - const request = { - text: 123 - }; - expect(() => validateDeIdentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_DEIDENTIFY); - }); - - // Test Options Validation - describe('Options Validation', () => { - const validRequest = { - text: 'valid text' - }; - - test('should throw error when entities is not an array', () => { - const options = { - getEntities: () => 'not-an-array' - }; - expect(() => validateDeIdentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_ENTITIES_IN_DEIDENTIFY); - }); - - test('should throw error when allowRegexList is not an array', () => { - const options = { - getEntities: () => null, - getAllowRegexList: () => 'not-an-array' - }; - expect(() => validateDeIdentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_ALLOW_REGEX_LIST); - }); - - test('should throw error when restrictRegexList is not an array', () => { - const options = { - getEntities: () => null, - getAllowRegexList: () => null, - getRestrictRegexList: () => 'not-an-array' - }; - expect(() => validateDeIdentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_RESTRICT_REGEX_LIST); - }); - - test('should throw error when tokenFormat is not TokenFormat instance', () => { - const options = { - getEntities: () => null, - getAllowRegexList: () => null, - getRestrictRegexList: () => null, - getTokenFormat: () => ({}) // not a TokenFormat instance - }; - expect(() => validateDeIdentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_TOKEN_FORMAT); - }); - - test('should throw error when transformations is not Transformations instance', () => { - const options = { - getEntities: () => null, - getAllowRegexList: () => null, - getRestrictRegexList: () => null, - getTokenFormat: () => null, - getTransformations: () => ({}) // not a Transformations instance - }; - expect(() => validateDeIdentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_TRANSFORMATIONS); - }); - }); - - // Test valid cases - test('should accept valid request without options', () => { - const request = { - text: 'valid text' - }; - expect(() => validateDeIdentifyTextRequest(request)).not.toThrow(); - }); - - test('should accept valid request with valid options', () => { - const request = { - text: 'valid text' - }; - const options = { - getEntities: () => ['PERSON', 'EMAIL'], - getAllowRegexList: () => ['^test', 'test$'], - getRestrictRegexList: () => ['^restricted', 'restricted$'], - getTokenFormat: () => new TokenFormat(), - getTransformations: () => new Transformations() - }; - expect(() => validateDeIdentifyTextRequest(request, options)).not.toThrow(); - }); - - // Test different log levels - test('should work with different log levels', () => { - const request = { - text: 'valid text' - }; - expect(() => validateDeIdentifyTextRequest(request, undefined, LogLevel.DEBUG)).not.toThrow(); - expect(() => validateDeIdentifyTextRequest(request, undefined, LogLevel.INFO)).not.toThrow(); - expect(() => validateDeIdentifyTextRequest(request, undefined, LogLevel.WARN)).not.toThrow(); - expect(() => validateDeIdentifyTextRequest(request, undefined, LogLevel.ERROR)).not.toThrow(); - }); - - // Test with various text content - test('should accept text with special characters', () => { - const request = { - text: 'Special chars: !@#$%^&*()' - }; - expect(() => validateDeIdentifyTextRequest(request)).not.toThrow(); - }); - - test('should accept multi-line text', () => { - const request = { - text: `Line 1 - Line 2 - Line 3` - }; - expect(() => validateDeIdentifyTextRequest(request)).not.toThrow(); - }); - - test('should accept text with multiple spaces', () => { - const request = { - text: 'Text with multiple spaces' - }; - expect(() => validateDeIdentifyTextRequest(request)).not.toThrow(); - }); -}); - -describe('validateReidentifyTextRequest', () => { - // Test for invalid text - test('should throw error when text is missing', () => { - const request = {}; - expect(() => validateReidentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_REIDENTIFY); - }); - - test('should throw error when text is empty', () => { - const request = { - text: '' - }; - expect(() => validateReidentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_REIDENTIFY); - }); - - test('should throw error when text is whitespace', () => { - const request = { - text: ' ' - }; - expect(() => validateReidentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_REIDENTIFY); - }); - - test('should throw error when text is not string', () => { - const request = { - text: 123 - }; - expect(() => validateReidentifyTextRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_TEXT_IN_REIDENTIFY); - }); - - // Test Options Validation - describe('ReidentifyTextOptions Validation', () => { - const validRequest = { - text: 'valid text with tokens' - }; - - test('should throw error when redactedEntities is not an array', () => { - const options = { - getRedactedEntities: () => 'not-an-array' - }; - expect(() => validateReidentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_REDACTED_ENTITIES_IN_REIDENTIFY); - }); - - test('should throw error when maskedEntities is not an array', () => { - const options = { - getRedactedEntities: () => null, - getMaskedEntities: () => 'not-an-array' - }; - expect(() => validateReidentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_MASKED_ENTITIES_IN_REIDENTIFY); - }); - - test('should throw error when plainTextEntities is not an array', () => { - const options = { - getRedactedEntities: () => null, - getMaskedEntities: () => null, - getPlainTextEntities: () => 'not-an-array' - }; - expect(() => validateReidentifyTextRequest(validRequest, options)) - .toThrow(SKYFLOW_ERROR_CODE.INVALID_PLAIN_TEXT_ENTITIES_IN_REIDENTIFY); - }); - - test('should accept valid arrays for all entity types', () => { - const options = { - getRedactedEntities: () => ['PERSON', 'EMAIL'], - getMaskedEntities: () => ['PHONE', 'SSN'], - getPlainTextEntities: () => ['ADDRESS'] - }; - expect(() => validateReidentifyTextRequest(validRequest, options)).not.toThrow(); - }); - - test('should accept empty arrays for entity types', () => { - const options = { - getRedactedEntities: () => [], - getMaskedEntities: () => [], - getPlainTextEntities: () => [] - }; - expect(() => validateReidentifyTextRequest(validRequest, options)).not.toThrow(); - }); - }); - - // Test valid cases - test('should accept valid request without options', () => { - const request = { - text: 'valid text with tokens' - }; - expect(() => validateReidentifyTextRequest(request)).not.toThrow(); - }); - - // Test different log levels - test('should work with different log levels', () => { - const request = { - text: 'valid text with tokens' - }; - expect(() => validateReidentifyTextRequest(request, undefined, LogLevel.DEBUG)).not.toThrow(); - expect(() => validateReidentifyTextRequest(request, undefined, LogLevel.INFO)).not.toThrow(); - expect(() => validateReidentifyTextRequest(request, undefined, LogLevel.WARN)).not.toThrow(); - expect(() => validateReidentifyTextRequest(request, undefined, LogLevel.ERROR)).not.toThrow(); - }); - - // Test with various text content - test('should accept text with special characters', () => { - const request = { - text: 'Text with special chars: !@#$%^&*()' - }; - expect(() => validateReidentifyTextRequest(request)).not.toThrow(); - }); - - test('should accept multi-line text', () => { - const request = { - text: `Line 1 - Line 2 - Line 3` - }; - expect(() => validateReidentifyTextRequest(request)).not.toThrow(); - }); - - test('should accept text with tokens', () => { - const request = { - text: 'Text with {{TOKEN_1}} and {{TOKEN_2}} embedded' - }; - expect(() => validateReidentifyTextRequest(request)).not.toThrow(); - }); - - // Test combinations of options - test('should accept request with mixed entity types', () => { - const request = { - text: 'Text with tokens' - }; - const options = { - getRedactedEntities: () => ['PERSON'], - getMaskedEntities: () => ['EMAIL'], - getPlainTextEntities: () => ['PHONE'] - }; - expect(() => validateReidentifyTextRequest(request, options)).not.toThrow(); - }); -}); - -describe('validateGetDetectRunRequest', () => { - // Test for missing runId - test('should throw error when runId is missing', () => { - const request = {}; - expect(() => validateGetDetectRunRequest(request)).toThrow(SKYFLOW_ERROR_CODE.EMPTY_RUN_ID); - }); - - // Test for invalid runId type - test('should throw error when runId is not a string', () => { - const request = { - runId: 123 // number instead of string - }; - expect(() => validateGetDetectRunRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_RUN_ID); - - const request2 = { - runId: {} // object instead of string - }; - expect(() => validateGetDetectRunRequest(request2)).toThrow(SKYFLOW_ERROR_CODE.INVALID_RUN_ID); - - const request3 = { - runId: true // boolean instead of string - }; - expect(() => validateGetDetectRunRequest(request3)).toThrow(SKYFLOW_ERROR_CODE.INVALID_RUN_ID); - }); - - // Test for empty runId - test('should throw error when runId is empty string', () => { - const request = { - runId: '' - }; - expect(() => validateGetDetectRunRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_RUN_ID); - }); - - // Test for whitespace runId - test('should throw error when runId contains only whitespace', () => { - const request = { - runId: ' ' - }; - expect(() => validateGetDetectRunRequest(request)).toThrow(SKYFLOW_ERROR_CODE.INVALID_RUN_ID); - }); - - // Test valid cases - test('should accept valid request', () => { - const request = { - runId: 'valid-run-id' - }; - expect(() => validateGetDetectRunRequest(request)).not.toThrow(); - }); - - // Test for different valid runId formats - test('should accept different valid runId formats', () => { - const request1 = { - runId: 'run-123' - }; - expect(() => validateGetDetectRunRequest(request1)).not.toThrow(); - - const request2 = { - runId: 'RUN_456' - }; - expect(() => validateGetDetectRunRequest(request2)).not.toThrow(); - - const request3 = { - runId: '789abc' - }; - expect(() => validateGetDetectRunRequest(request3)).not.toThrow(); - }); - - // Test different log levels - test('should work with different log levels', () => { - const request = { - runId: 'valid-run-id' - }; - expect(() => validateGetDetectRunRequest(request, LogLevel.DEBUG)).not.toThrow(); - expect(() => validateGetDetectRunRequest(request, LogLevel.INFO)).not.toThrow(); - expect(() => validateGetDetectRunRequest(request, LogLevel.WARN)).not.toThrow(); - expect(() => validateGetDetectRunRequest(request, LogLevel.ERROR)).not.toThrow(); - }); - - // Test special characters in runId - test('should accept runId with special characters', () => { - const request = { - runId: 'run-id_123.456@test' - }; - expect(() => validateGetDetectRunRequest(request)).not.toThrow(); - }); - -}); - describe('validateInvokeConnectionRequest', () => { // Test for null/undefined request test('should throw error when request is null/undefined', () => { @@ -4678,59 +3963,6 @@ describe('additional validation branch coverage', () => { })).not.toThrow(); }); - test('validateDeidentifyFileRequest accepts fully populated valid options', () => { - jest.spyOn(fs, 'existsSync').mockReturnValue(true); - jest.spyOn(fs, 'lstatSync').mockReturnValue({ isDirectory: () => true }); - const file = new File(['content'], 'sample.txt', { type: 'text/plain' }); - const request = new DeidentifyFileRequest({ file }); - const options = new DeidentifyFileOptions(); - const tokenFormat = new TokenFormat(); - tokenFormat.setDefault('entity_unique_counter'); - const transformations = new Transformations(); - transformations.setShiftDays({ max: 5, min: 1, entities: ['DATE'] }); - const bleep = new Bleep(); - bleep.setGain(0.5); - bleep.setFrequency(440); - options.setWaitTime(10); - options.setOutputDirectory('/valid/output/dir'); - options.setEntities(['NAME']); - options.setAllowRegexList(['allow']); - options.setRestrictRegexList(['restrict']); - options.setTokenFormat(tokenFormat); - options.setTransformations(transformations); - options.setOutputProcessedImage(true); - options.setOutputOcrText(true); - options.setMaskingMethod(MaskingMethod.Blur); - options.setPixelDensity(300); - options.setMaxResolution(1920); - options.setOutputProcessedAudio(true); - options.setOutputTranscription('transcription'); - options.setBleep(bleep); - expect(() => validateDeidentifyFileRequest(request, options)).not.toThrow(); - }); - - test('validateDeIdentifyTextRequest accepts valid options', () => { - const options = new (require('../../src/vault/model/options/deidentify-text').default)(); - const tokenFormat = new TokenFormat(); - const transformations = new Transformations(); - transformations.setShiftDays({ max: 3, min: 1, entities: ['DATE'] }); - options.setEntities(['NAME']); - options.setAllowRegexList(['a']); - options.setRestrictRegexList(['b']); - options.setTokenFormat(tokenFormat); - options.setTransformations(transformations); - expect(() => validateDeIdentifyTextRequest({ text: 'hello' }, options)).not.toThrow(); - }); - - test('validateReidentifyTextRequest accepts valid entity lists', () => { - const ReidentifyTextOptions = require('../../src/vault/model/options/reidentify-text').default; - const options = new ReidentifyTextOptions(); - options.setRedactedEntities(['NAME']); - options.setMaskedEntities(['EMAIL']); - options.setPlainTextEntities(['PHONE']); - expect(() => validateReidentifyTextRequest({ text: 'redacted' }, options)).not.toThrow(); - }); - test('validateInvokeConnectionRequest accepts string body with content type', () => { expect(() => validateInvokeConnectionRequest({ method: 'POST', diff --git a/skyvault/test/vault/client/client.test.js b/skyvault/test/vault/client/client.test.js index 24511701..189d24d8 100644 --- a/skyvault/test/vault/client/client.test.js +++ b/skyvault/test/vault/client/client.test.js @@ -5,15 +5,11 @@ import { Tokens } from '../../../src/ _generated_/rest/api/resources/tokens/clie import { AuthType, LogLevel, TYPES } from 'skyflow-common'; import { isExpired } from 'skyflow-common'; import { SkyflowError } from 'skyflow-common'; -import { Strings } from '../../../src/ _generated_/rest/api/resources/strings/client/Client'; -import { Files } from '../../../src/ _generated_/rest/api/resources/files/client/Client'; jest.mock('../../../src/ _generated_/rest'); jest.mock('../../../src/ _generated_/rest/api/resources/records/client/Client'); jest.mock('../../../src/ _generated_/rest/api/resources/query/client/Client'); jest.mock('../../../src/ _generated_/rest/api/resources/tokens/client/Client'); -jest.mock('../../../src/ _generated_/rest/api/resources/strings/client/Client'); -jest.mock('../../../src/ _generated_/rest/api/resources/files/client/Client'); jest.mock('skyflow-common', () => { const actual = jest.requireActual('skyflow-common'); return { @@ -100,11 +96,6 @@ describe('VaultClient', () => { expect(Query).toHaveBeenCalled(); }); - test('should initialize Strings for DETECT', () => { - vaultClient.initAPI(authInfo, TYPES.DEIDENTIFY_TEXT); - expect(Strings).toHaveBeenCalled(); - }); - test('should not initialize API for unsupported type', () => { vaultClient.initAPI(authInfo, 'UNSUPPORTED_TYPE'); expect(Records).not.toHaveBeenCalled(); @@ -1080,18 +1071,6 @@ describe('VaultClient', () => { expect(Query).toHaveBeenCalledTimes(1); }); - test('Strings client is created only once across DEIDENTIFY_TEXT and REIDENTIFY_TEXT calls', () => { - vaultClient.initAPI(authInfo, TYPES.DEIDENTIFY_TEXT); - vaultClient.initAPI(authInfo, TYPES.REIDENTIFY_TEXT); - expect(Strings).toHaveBeenCalledTimes(1); - }); - - test('Files client is created only once across DEIDENTIFY_FILE and DETECT_RUN calls', () => { - vaultClient.initAPI(authInfo, TYPES.DEIDENTIFY_FILE); - vaultClient.initAPI(authInfo, TYPES.DETECT_RUN); - expect(Files).toHaveBeenCalledTimes(1); - }); - test('each API type gets its own client (INSERT + DETOKENIZE + QUERY each create once)', () => { vaultClient.initAPI(authInfo, TYPES.INSERT); vaultClient.initAPI(authInfo, TYPES.DETOKENIZE); @@ -1196,20 +1175,6 @@ describe('VaultClient', () => { expect(typeof opts.token).toBe('function'); expect(typeof opts.baseUrl).toBe('function'); }); - - test('Strings constructor also receives supplier functions', () => { - vaultClient.initAPI(authInfo, TYPES.DEIDENTIFY_TEXT); - const opts = Strings.mock.calls[0][0]; - expect(typeof opts.token).toBe('function'); - expect(typeof opts.baseUrl).toBe('function'); - }); - - test('Files constructor also receives supplier functions', () => { - vaultClient.initAPI(authInfo, TYPES.DEIDENTIFY_FILE); - const opts = Files.mock.calls[0][0]; - expect(typeof opts.token).toBe('function'); - expect(typeof opts.baseUrl).toBe('function'); - }); }); }); diff --git a/skyvault/test/vault/controller/detect.test.js b/skyvault/test/vault/controller/detect.test.js index 5d55678d..e298a7fa 100644 --- a/skyvault/test/vault/controller/detect.test.js +++ b/skyvault/test/vault/controller/detect.test.js @@ -1,2205 +1,80 @@ import DetectController from '../../../src/vault/controller/detect'; -import { validateDeidentifyFileRequest, validateDeIdentifyTextRequest, validateGetDetectRunRequest, validateReidentifyTextRequest } from '../../../src/utils/validations'; -import DeidentifyTextResponse from '../../../src/vault/model/response/deidentify-text'; -import ReidentifyTextResponse from '../../../src/vault/model/response/reidentify-text'; -import DeidentifyFileRequest from '../../../src/vault/model/request/deidentify-file'; -import DeidentifyFileOptions from '../../../src/vault/model/options/deidentify-file'; -import TokenFormat from '../../../src/vault/model/options/deidentify-text/token-format'; -import Transformations from '../../../src/vault/model/options/deidentify-text/transformations'; -import { Bleep } from '../../../src/vault/model/options/deidentify-file/bleep-audio'; -import { DETECT_STATUS, ENCODING_TYPE, TYPES } from 'skyflow-common'; -import fs from 'fs'; - -function createFullDeidentifyFileOptions() { - const options = new DeidentifyFileOptions(); - const tokenFormat = new TokenFormat(); - tokenFormat.setDefault('entity_unique_counter'); - tokenFormat.setEntityUniqueCounter(['NAME']); - tokenFormat.setEntityOnly(['EMAIL']); - - const transformations = new Transformations(); - transformations.setShiftDays({ max: 10, min: 2, entities: ['DATE'] }); - - const bleep = new Bleep(); - bleep.setGain(0.5); - bleep.setFrequency(440); - bleep.setStartPadding(0.1); - bleep.setStopPadding(0.2); - - options.setEntities(['NAME', 'EMAIL']); - options.setAllowRegexList(['allow-regex']); - options.setRestrictRegexList(['restrict-regex']); - options.setTokenFormat(tokenFormat); - options.setTransformations(transformations); - options.setOutputTranscription('transcription'); - options.setOutputProcessedAudio(true); - options.setOutputProcessedImage(true); - options.setOutputOcrText(true); - options.setMaskingMethod('blackbox'); - options.setMaxResolution(1920); - options.setPixelDensity(300); - options.setBleep(bleep); - return options; -} +import VaultClient from '../../../src/vault/client'; +import { DeidentifySourceType, ReidentifySourceType } from '../../../src/vault/detect/types'; +import { + AuthType, + BaseDetectController, + EntityType, + GetRunRequest, + LogLevel, + RedactionType, + ReidentifyFileRequest, + getBearerToken, +} from 'skyflow-common'; jest.mock('skyflow-common', () => { const actual = jest.requireActual('skyflow-common'); return { ...actual, printLog: jest.fn(), - parameterizedString: jest.fn(), - removeSDKVersion: jest.fn(), - generateSDKMetrics: jest.fn().mockReturnValue({ sdk: 'metrics' }), - getBearerToken: jest.fn().mockResolvedValue(Promise.resolve('your-bearer-token')), + getBearerToken: jest.fn(), }; }); -jest.mock('../../../src/utils/validations', () => ({ - validateDeIdentifyTextRequest: jest.fn(), - validateReidentifyTextRequest: jest.fn(), - validateDeidentifyFileRequest: jest.fn(), - validateGetDetectRunRequest: jest.fn(), -})); - -describe('DetectController', () => { - let mockVaultClient; - let detectController; - - beforeEach(() => { - mockVaultClient = { - getLogLevel: jest.fn().mockReturnValue('DEBUG'), - getCredentials: jest.fn().mockReturnValue({}), - initAPI: jest.fn(), - stringsAPI: { - deidentifyString: jest.fn(), - reidentifyString: jest.fn(), - }, - filesAPI: { - deidentifyPdf: jest.fn(), - deidentifyAudio: jest.fn(), - getRun: jest.fn(), - }, - failureResponse: jest.fn().mockRejectedValue(new Error('API error')), - vaultId: 'vault123', - }; - jest.clearAllMocks(); - }); - - test('should initialize DetectController and call printLog with correct parameters', () => { - detectController = new DetectController(mockVaultClient); - expect(detectController).toBeInstanceOf(DetectController); - }); -}); - -describe('deidentifyText', () => { - - let mockVaultClient; - let detectController; - - beforeEach(() => { - mockVaultClient = { - getLogLevel: jest.fn().mockReturnValue('DEBUG'), - getCredentials: jest.fn().mockReturnValue({}), - initAPI: jest.fn(), - stringsAPI: { - deidentifyString: jest.fn(), - reidentifyString: jest.fn(), - }, - failureResponse: jest.fn().mockRejectedValue(new Error('API error')), - vaultId: 'vault123', - }; - detectController = new DetectController(mockVaultClient); - jest.clearAllMocks(); - }); - - test('should successfully deidentify text', async () => { - const mockRequest = { - text: 'Sensitive data to deidentify', - }; - const mockOptions = { - getEntities: jest.fn().mockReturnValue(['NAME', 'EMAIL']), - getAllowRegexList: jest.fn().mockReturnValue(['regex1']), - getRestrictRegexList: jest.fn().mockReturnValue(['regex2']), - getTransformations: jest.fn().mockReturnValue({ - getShiftDays: jest.fn().mockReturnValue({ - max: 10, - min: 5, - entities: ['DATE'], - }), - }), - getTokenFormat: jest.fn().mockReturnValue({ - getDefault: jest.fn().mockReturnValue('default'), - getVaultToken: jest.fn().mockReturnValue('vault-token'), - getEntityUniqueCounter: jest.fn().mockReturnValue('entity-unique-counter'), - getEntityOnly: jest.fn().mockReturnValue('entity-only'), - }) - }; - - const mockResponseData = { - processed_text: 'Processed text', - entities: [ - { - token: 'token1', - value: 'value1', - location: { - start_index: 0, - end_index: 5, - start_index_processed: 0, - end_index_processed: 5, - }, - entity_type: 'NAME', - entity_scores: [0.9], - }, - ], - word_count: 5, - character_count: 30, - }; - - mockVaultClient.stringsAPI.deidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: mockResponseData, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - const response = await detectController.deidentifyText(mockRequest, mockOptions); - - expect(validateDeIdentifyTextRequest).toHaveBeenCalledWith(mockRequest, mockOptions, 'DEBUG'); - expect(mockVaultClient.stringsAPI.deidentifyString).toHaveBeenCalledWith( - expect.objectContaining({ - text: mockRequest.text, - vault_id: 'vault123', - entity_types: ['NAME', 'EMAIL'], - allow_regex: ['regex1'], - restrict_regex: ['regex2'], - transformations: { - shift_dates: { - max_days: 10, - min_days: 5, - entity_types: ['DATE'], - }, - }, - }) - ); - expect(response).toBeInstanceOf(DeidentifyTextResponse); - expect(response.processedText).toBe('Processed text'); - expect(response.entities).toHaveLength(1); - expect(response.wordCount).toBe(5); - expect(response.charCount).toBe(30); - expect(response.errors).toBeNull(); - }); - - test('should map entities with missing optional fields', async () => { - validateDeIdentifyTextRequest.mockImplementation(() => {}); - const mockRequest = { text: 'text with entity' }; - - mockVaultClient.stringsAPI.deidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { - processed_text: 'processed', - entities: [{}], - word_count: 1, - character_count: 9, - }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - const response = await detectController.deidentifyText(mockRequest); - expect(response.entities[0]).toEqual({ - token: undefined, - value: undefined, - textIndex: { start: undefined, end: undefined }, - processedIndex: { start: undefined, end: undefined }, - entity: undefined, - scores: undefined, - }); - }); - - test('should map entities with partial location fields', async () => { - validateDeIdentifyTextRequest.mockImplementation(() => {}); - const mockRequest = { text: 'partial entity fields' }; - - mockVaultClient.stringsAPI.deidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { - processed_text: 'processed', - entities: [ - { token: 'tok', location: { start_index: 0, end_index: 1 } }, - { value: 'val', location: { start_index_processed: 2, end_index_processed: 3 } }, - ], - word_count: 2, - character_count: 20, - }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - const response = await detectController.deidentifyText(mockRequest); - expect(response.entities[0].textIndex).toEqual({ start: 0, end: 1 }); - expect(response.entities[0].processedIndex).toEqual({ - start: undefined, - end: undefined, - }); - expect(response.entities[1].processedIndex).toEqual({ start: 2, end: 3 }); - expect(response.entities[1].textIndex).toEqual({ - start: undefined, - end: undefined, - }); - }); - - test('should handle undefined records in deidentify text response', async () => { - validateDeIdentifyTextRequest.mockImplementation(() => {}); - const mockRequest = { text: 'no records payload' }; - - mockVaultClient.stringsAPI.deidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: undefined, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - const response = await detectController.deidentifyText(mockRequest); - expect(response.processedText).toBeUndefined(); - expect(response.entities).toEqual([]); - }); - - test('should map null entity entries safely', async () => { - validateDeIdentifyTextRequest.mockImplementation(() => {}); - const mockRequest = { text: 'null entity entry' }; - - mockVaultClient.stringsAPI.deidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { - processed_text: 'processed', - entities: [null], - word_count: 1, - character_count: 9, - }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - const response = await detectController.deidentifyText(mockRequest); - expect(response.entities[0]).toEqual({ - token: undefined, - value: undefined, - textIndex: { start: undefined, end: undefined }, - processedIndex: { start: undefined, end: undefined }, - entity: undefined, - scores: undefined, - }); - }); - - test('should handle validation errors', async () => { - const mockRequest = { - text: 'Sensitive data to deidentify', - }; - const mockOptions = {}; - - validateDeIdentifyTextRequest.mockImplementation(() => { - throw new Error('Validation error'); - }); - - await expect(detectController.deidentifyText(mockRequest, mockOptions)).rejects.toThrow( - 'Validation error' - ); - expect(validateDeIdentifyTextRequest).toHaveBeenCalledWith(mockRequest, mockOptions, 'DEBUG'); - expect(mockVaultClient.stringsAPI.deidentifyString).not.toHaveBeenCalled(); - }); - - test('should handle API errors during deidentifyText', async () => { - jest.mock('../../../src/utils/validations', () => ({ - validateDeIdentifyTextRequest: jest.fn(), - validateReidentifyTextRequest: jest.fn(), - })); - validateDeIdentifyTextRequest.mockImplementation(() => { - // No validation error - }); - const mockRequest = { - text: 'Sensitive data to deidentify', - }; - const mockOptions = { - getEntities: jest.fn().mockReturnValue([]), // Return an empty array for entities - getAllowRegexList: jest.fn().mockReturnValue([]), // Return an empty array for allow regex - getRestrictRegexList: jest.fn().mockReturnValue([]), // Return an empty array for restrict regex - getTransformations: jest.fn().mockReturnValue({ - getShiftDays: jest.fn().mockReturnValue({ - max: 10, - min: 5, - entities: ['DATE'], - }), - }), - getTokenFormat: jest.fn().mockReturnValue({ - getDefault: jest.fn().mockReturnValue('default'), - getVaultToken: jest.fn().mockReturnValue('vault-token'), - getEntityUniqueCounter: jest.fn().mockReturnValue('entity-unique-counter'), - getEntityOnly: jest.fn().mockReturnValue('entity-only'), - }) - }; - - mockVaultClient.stringsAPI.deidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockRejectedValue(new Error('API error')), - })); - - await expect(detectController.deidentifyText(mockRequest, mockOptions)).rejects.toThrow( - 'API error' - ); - expect(mockVaultClient.stringsAPI.deidentifyString).toHaveBeenCalled(); - }); - - // Lines 418-432: buildDeidentifyTextRequest options?.xxx false branches (options undefined) - test('should deidentify text without options (covers options?.xxx false branches)', async () => { - validateDeIdentifyTextRequest.mockImplementation(() => {}); - const mockRequest = { text: 'text to deidentify' }; - - mockVaultClient.stringsAPI.deidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { processed_text: 'processed text', entities: [], word_count: 1, character_count: 10 }, - rawResponse: null, // covers rawResponse?.headers false branch at line 389 - }), - })); - - const response = await detectController.deidentifyText(mockRequest); - expect(response).toBeInstanceOf(DeidentifyTextResponse); - expect(response.processedText).toBe('processed text'); - }); -}); - -describe('reidentifyText', () => { - let mockVaultClient; - let detectController; - - beforeEach(() => { - mockVaultClient = { - getLogLevel: jest.fn().mockReturnValue('DEBUG'), - getCredentials: jest.fn().mockReturnValue({}), - initAPI: jest.fn(), - stringsAPI: { - deidentifyString: jest.fn(), - reidentifyString: jest.fn(), - }, - failureResponse: jest.fn().mockRejectedValue(new Error('API error')), - vaultId: 'vault123', - }; - detectController = new DetectController(mockVaultClient); - jest.clearAllMocks(); - }); - test('should successfully reidentify text', async () => { - const mockRequest = { - text: 'Redacted text to reidentify', - }; - const mockOptions = { - getRedactedEntities: jest.fn().mockReturnValue(['NAME']), - getMaskedEntities: jest.fn().mockReturnValue(['EMAIL']), - getPlainTextEntities: jest.fn().mockReturnValue(['PHONE']), - }; - - const mockResponseData = { - text: 'Reidentified text', - }; - - mockVaultClient.stringsAPI.reidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: mockResponseData, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - const response = await detectController.reidentifyText(mockRequest, mockOptions); - - expect(validateReidentifyTextRequest).toHaveBeenCalledWith(mockRequest, mockOptions, 'DEBUG'); - expect(mockVaultClient.stringsAPI.reidentifyString).toHaveBeenCalledWith( - expect.objectContaining({ - text: mockRequest.text, - vault_id: 'vault123', - format: { - redacted: ['NAME'], - masked: ['EMAIL'], - plaintext: ['PHONE'], - }, - }) - ); - expect(response).toBeInstanceOf(ReidentifyTextResponse); - expect(response.processedText).toBe('Reidentified text'); - }); - - test('should handle validation errors', async () => { - const mockRequest = { - text: 'Redacted text to reidentify', - }; - const mockOptions = { - getRedactedEntities: jest.fn().mockReturnValue(['NAME']), - getMaskedEntities: jest.fn().mockReturnValue(['EMAIL']), - getPlainTextEntities: jest.fn().mockReturnValue(['PHONE']), - }; - - validateReidentifyTextRequest.mockImplementation(() => { - throw new Error('Validation error'); - }); - - await expect(detectController.reidentifyText(mockRequest, mockOptions)).rejects.toThrow( - 'Validation error' - ); - expect(validateReidentifyTextRequest).toHaveBeenCalledWith(mockRequest, mockOptions, 'DEBUG'); - expect(mockVaultClient.stringsAPI.reidentifyString).not.toHaveBeenCalled(); - }); - - test('should handle API errors during reidentifyText', async () => { - jest.mock('../../../src/utils/validations', () => ({ - validateDeIdentifyTextRequest: jest.fn(), - validateReidentifyTextRequest: jest.fn(), - })); - validateReidentifyTextRequest.mockImplementation(() => { - // No validation error - }); - const mockRequest = { - text: 'Redacted text to reidentify', - }; - const mockOptions = { - getRedactedEntities: jest.fn().mockReturnValue(['NAME']), - getMaskedEntities: jest.fn().mockReturnValue(['EMAIL']), - getPlainTextEntities: jest.fn().mockReturnValue(['PHONE']), - }; - - mockVaultClient.stringsAPI.reidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockRejectedValue(new Error('API error')), - })); - - await expect(detectController.reidentifyText(mockRequest, mockOptions)).rejects.toThrow( - 'API error' - ); - expect(mockVaultClient.stringsAPI.reidentifyString).toHaveBeenCalled(); - }); - - test('should reidentify text without options (covers options?.xxx false branches)', async () => { - validateReidentifyTextRequest.mockImplementation(() => {}); - const mockRequest = { text: 'redacted text' }; - - mockVaultClient.stringsAPI.reidentifyString.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { text: 'plain text' }, - rawResponse: null, - }), - })); - - const response = await detectController.reidentifyText(mockRequest); - expect(response).toBeInstanceOf(ReidentifyTextResponse); - expect(response.processedText).toBe('plain text'); - }); -}); - -describe('getDetectRun', () => { - let mockVaultClient; - let detectController; - - beforeEach(() => { - mockVaultClient = { - getLogLevel: jest.fn().mockReturnValue('DEBUG'), - getCredentials: jest.fn().mockReturnValue({}), - initAPI: jest.fn(), - filesAPI: { - getRun: jest.fn(), - }, - failureResponse: jest.fn().mockRejectedValue(new Error('API error')), - vaultId: 'vault123', - }; - detectController = new DetectController(mockVaultClient); - jest.clearAllMocks(); - }); - - test('should successfully get detect run and parse response', async () => { - jest.mock('../../../src/utils/validations', () => ({ - validateGetDetectRunRequest: jest.fn(), - })); - validateGetDetectRunRequest.mockImplementation(() => { - // No validation error - }); - const mockRequest = { runId: 'mockRunId' }; - const mockResponseData = { - output: [ - { - processedFile: 'mockProcessedFile', - processedFileType: 'text', - processedFileExtension: 'txt', - }, - ], - wordCharacterCount: { - wordCount: 100, - characterCount: 500, - }, - size: 1024, - duration: 0, - pages: 1, - slides: 0, - run_id: 'mockRunId', - }; - - mockVaultClient.filesAPI.getRun.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: mockResponseData, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - const result = await detectController.getDetectRun(mockRequest); - - expect(mockVaultClient.filesAPI.getRun).toHaveBeenCalledWith( - mockRequest.runId, - { vault_id: 'vault123' } - ); - }); - - test('should prefer runId from response data over request runId', async () => { - validateGetDetectRunRequest.mockImplementation(() => {}); - const mockRequest = { runId: 'request-run-id' }; - const mockResponseData = { - runId: 'response-run-id', - status: 'SUCCESS', - output: [ - { - processedFile: Buffer.from('file').toString(ENCODING_TYPE.BASE64), - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }, - ], - wordCharacterCount: { wordCount: 1, characterCount: 10 }, - size: 100, - duration: 0, - pages: 1, - slides: 0, - }; - - mockVaultClient.filesAPI.getRun.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: mockResponseData, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - const result = await detectController.getDetectRun(mockRequest); - expect(result.runId).toBe('response-run-id'); - }); - - test('should handle validation errors', async () => { - const mockRequest = { runId: 'mockRunId' }; - validateGetDetectRunRequest.mockImplementation(() => { - throw new Error('Validation error'); - }); - - await expect(detectController.getDetectRun(mockRequest)).rejects.toThrow('Validation error'); - expect(mockVaultClient.filesAPI.getRun).not.toHaveBeenCalled(); - }); - - test('should handle API errors during getDetectRun', async () => { - jest.mock('../../../src/utils/validations', () => ({ - validateGetDetectRunRequest: jest.fn(), - })); - validateGetDetectRunRequest.mockImplementation(() => { - // No validation error - }); - const mockRequest = { runId: 'mockRunId' }; - mockVaultClient.filesAPI.getRun.mockImplementation(() => ({ - withRawResponse: jest.fn().mockRejectedValue(new Error('API error')), - })); - - await expect(detectController.getDetectRun(mockRequest)).rejects.toThrow('API error'); - }); - - // Lines 580-583: catch block — if (error instanceof Error) false branch - test('should handle non-Error thrown in getDetectRun catch block', async () => { - validateGetDetectRunRequest.mockImplementation(() => { - throw 'non-error string exception'; - }); - const mockRequest = { runId: 'mockRunId' }; - await expect(detectController.getDetectRun(mockRequest)).rejects.toBe('non-error string exception'); - }); -}); - -describe('deidentifyFile', () => { - let mockVaultClient; - let detectController; - - beforeEach(() => { - jest.useFakeTimers(); - mockVaultClient = { - getLogLevel: jest.fn().mockReturnValue('DEBUG'), - getCredentials: jest.fn().mockReturnValue({}), - initAPI: jest.fn(), - filesAPI: { - deidentifyText: jest.fn(), - deidentifyAudio: jest.fn(), - deidentifyPdf: jest.fn(), - deidentifyImage: jest.fn(), - deidentifyPresentation: jest.fn(), - deidentifySpreadsheet: jest.fn(), - deidentifyStructuredText: jest.fn(), - deidentifyDocument: jest.fn(), - deidentifyFile: jest.fn(), - getRun: jest.fn(), - }, - failureResponse: jest.fn().mockRejectedValue(new Error('API error')), - vaultId: 'vault123', - }; - detectController = new DetectController(mockVaultClient); - jest.clearAllMocks(); - }); - - afterEach(() => { - jest.useRealTimers(); - }); - - test('should successfully deidentify a PDF file and poll until SUCCESS', async () => { - // Arrange - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setPixelDensity(300); - options.setMaxResolution(2000); - - // Mock PDF deidentify API to return a run_id - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'run123' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - // Polling: IN_PROGRESS twice, then SUCCESS - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'mockProcessedFile', - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }, - ], - wordCharacterCount: { - wordCount: 10, - characterCount: 100, - }, - size: 2048, - duration: 0, - pages: 2, - slides: 0, - run_id: 'run123', - }); - - // Act - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - - // Fast-forward all timers and flush all microtasks - await jest.runAllTimersAsync(); - - const result = await promise; - - // Assert - expect(mockVaultClient.filesAPI.deidentifyPdf).toHaveBeenCalled(); - expect(mockVaultClient.filesAPI.getRun).toHaveBeenCalledTimes(3); - expect(result.fileBase64).toBe('mockProcessedFile'); - expect(result.type).toBe('pdf'); - expect(result.extension).toBe('pdf'); - expect(result.wordCount).toBe(10); - expect(result.charCount).toBe(100); - expect(result.sizeInKb).toBe(2048); - expect(result.pageCount).toBe(2); - expect(result.status).toBe('SUCCESS'); - expect(result.errors).toBeNull(); - }); - - test('should successfully deidentify a PDF file using file path', async () => { - const filePath = 'test/mockData/pii.pdf'; - const deidentifyFileReq = new DeidentifyFileRequest({filePath}); - const options = new DeidentifyFileOptions(); - options.setPixelDensity(300); - options.setMaxResolution(2000); - - // Mock fs.promises.readFile so fake timers don't block on real I/O - jest.spyOn(fs.promises, 'readFile').mockResolvedValue(Buffer.from('dummy pdf content')); - - // Mock PDF deidentify API to return a run_id - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'run123' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - // Polling: IN_PROGRESS twice, then SUCCESS - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'mockProcessedFile', - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }, - ], - wordCharacterCount: { - wordCount: 10, - characterCount: 100, - }, - size: 2048, - duration: 0, - pages: 2, - slides: 0, - run_id: 'run123', - }); - - // Act - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - - // Fast-forward all timers and flush all microtasks - await jest.runAllTimersAsync(); - - const result = await promise; - - // Assert - expect(mockVaultClient.filesAPI.deidentifyPdf).toHaveBeenCalled(); - expect(mockVaultClient.filesAPI.getRun).toHaveBeenCalledTimes(3); - expect(result.fileBase64).toBe('mockProcessedFile'); - expect(result.type).toBe('pdf'); - expect(result.extension).toBe('pdf'); - expect(result.wordCount).toBe(10); - expect(result.charCount).toBe(100); - expect(result.sizeInKb).toBe(2048); - expect(result.pageCount).toBe(2); - expect(result.status).toBe('SUCCESS'); - - // Assert file object - expect(result.file).toBeInstanceOf(File); - expect(result.file.name).toBe('deidentified.pdf'); - }); - - test('should reject for PDF if polling returns FAILED', async () => { - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest(file); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'run123' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValue({ status: 'FAILED', message: 'Processing failed' }); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow(); - }); - - test('should reject for PDF if polling throws error', async () => { - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'run123' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockRejectedValue(new Error('Polling error')); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow('Polling error'); - }); - - test('should reject if deidentifyPdf throws error', async () => { - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockRejectedValue(new Error('API error')), - })); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow('API error'); - }); - - test('should successfully deidentify an audio file and poll until SUCCESS', async () => { - const file = new File(['audio content'], 'test.mp3'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyAudio.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'audioRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-456') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'audioProcessedFile', - processedFileType: 'mp3', - processedFileExtension: 'mp3', - }, - ], - wordCharacterCount: { - wordCount: 5, - characterCount: 50, - }, - size: 512, - duration: 10, - pages: 0, - slides: 0, - run_id: 'audioRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(mockVaultClient.filesAPI.deidentifyAudio).toHaveBeenCalled(); - }); - - test('should successfully deidentify an image file and poll until SUCCESS', async () => { - const file = new File(['image content'], 'test.png', { type: 'image/png' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyImage.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'imgRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-789') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'imgProcessedFile', - processedFileType: 'png', - processedFileExtension: 'png', - }, - ], - wordCharacterCount: { - wordCount: 2, - characterCount: 20, - }, - size: 256, - duration: 0, - pages: 0, - slides: 0, - run_id: 'imgRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(mockVaultClient.filesAPI.deidentifyImage).toHaveBeenCalled(); - expect(mockVaultClient.filesAPI.getRun).toHaveBeenCalledTimes(3); - }); - - test('should successfully deidentify a spreadsheet file and poll until SUCCESS', async () => { - const file = new File(['spreadsheet content'], 'test.xlsx', { type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifySpreadsheet.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'sheetRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-321') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'sheetProcessedFile', - processedFileType: 'xlsx', - processedFileExtension: 'xlsx', - }, - ], - wordCharacterCount: { - wordCount: 8, - characterCount: 80, - }, - size: 1024, - duration: 0, - pages: 0, - slides: 0, - run_id: 'sheetRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(mockVaultClient.filesAPI.deidentifySpreadsheet).toHaveBeenCalled(); - expect(result.fileBase64).toBe('sheetProcessedFile'); - expect(result.type).toBe('xlsx'); - expect(result.extension).toBe('xlsx'); - expect(result.wordCount).toBe(8); - expect(result.charCount).toBe(80); - expect(result.sizeInKb).toBe(1024); - expect(result.status).toBe('SUCCESS'); - }); - - test('should successfully deidentify a ppt file and poll until SUCCESS', async () => { - const file = new File(['ppt content'], 'test.pptx', { type: 'application/vnd.openxmlformats-officedocument.presentationml.presentation' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPresentation.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'pptRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-654') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'pptProcessedFile', - processedFileType: 'pptx', - processedFileExtension: 'pptx', - }, - ], - wordCharacterCount: { - wordCount: 3, - characterCount: 30, - }, - size: 2048, - duration: 0, - pages: 0, - slides: 5, - run_id: 'pptRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(result.fileBase64).toBe('pptProcessedFile'); - expect(result.type).toBe('pptx'); - expect(result.extension).toBe('pptx'); - expect(result.wordCount).toBe(3); - expect(result.charCount).toBe(30); - expect(result.sizeInKb).toBe(2048); - expect(result.slideCount).toBe(5); - expect(result.status).toBe('SUCCESS'); - }); - - test('should successfully deidentify a structured text file and poll until SUCCESS', async () => { - const file = new File(['json content'], 'test.json', { type: 'application/json' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyStructuredText.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'jsonRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-987') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'jsonProcessedFile', - processedFileType: 'json', - processedFileExtension: 'json', - }, - ], - wordCharacterCount: { - wordCount: 6, - characterCount: 60, - }, - size: 512, - duration: 0, - pages: 0, - slides: 0, - run_id: 'jsonRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(result.fileBase64).toBe('jsonProcessedFile'); - expect(result.type).toBe('json'); - expect(result.extension).toBe('json'); - expect(result.wordCount).toBe(6); - expect(result.charCount).toBe(60); - expect(result.sizeInKb).toBe(512); - expect(result.status).toBe('SUCCESS'); - }); - - test('should successfully deidentify a document file and poll until SUCCESS', async () => { - const file = new File(['doc content'], 'test.docx', { type: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyDocument.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'docRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-111') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'docProcessedFile', - processedFileType: 'docx', - processedFileExtension: 'docx', - }, - ], - wordCharacterCount: { - wordCount: 7, - characterCount: 70, - }, - size: 1024, - duration: 0, - pages: 1, - slides: 0, - run_id: 'docRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(result.fileBase64).toBe('docProcessedFile'); - expect(result.type).toBe('docx'); - expect(result.extension).toBe('docx'); - expect(result.wordCount).toBe(7); - expect(result.charCount).toBe(70); - expect(result.sizeInKb).toBe(1024); - expect(result.pageCount).toBe(1); - expect(result.status).toBe('SUCCESS'); - }); - - test('should successfully deidentify a text file and poll until SUCCESS', async () => { - // 1. Data Setup - const file = new File(['doc content'], 'test.txt', { type: 'text/plain' }); - const deidentifyFileReq = new DeidentifyFileRequest({ file }); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/mock/output/directory'); - - // 2. Mock File System (source uses fs.promises — spy on async methods) - const mkdirSpy = jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - const writeFileSpy = jest.spyOn(fs.promises, 'writeFile').mockResolvedValue(undefined); - - // 3. Mock deidentifyFile (The specific method causing the TypeError) - mockVaultClient.filesAPI.deidentifyFile = jest.fn().mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'docRunId' }, - rawResponse: { - headers: { - get: jest.fn().mockReturnValue('request-id-111') - } - }, - }), - })); - - mockVaultClient.filesAPI.getRun = jest.fn() - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: Buffer.from('textProcessedFile').toString('base64'), - processedFileType: 'text', - processedFileExtension: 'txt', - }, - ], - wordCharacterCount: { - wordCount: 7, - characterCount: 70, - }, - size: 1024, - duration: 0, - pages: 1, - slides: 0, - run_id: 'docRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - - // Fast-forward through the polling intervals - await jest.runAllTimersAsync(); - - const result = await promise; - - // 6. Assertions - expect(result.extension).toBe('txt'); - expect(result.wordCount).toBe(7); - expect(result.status).toBe('SUCCESS'); - - // Verify file system interactions - expect(mkdirSpy).toHaveBeenCalledWith('/mock/output/directory', { recursive: true }); - expect(writeFileSpy).toHaveBeenCalledWith( - expect.stringContaining('processed-test.txt'), - expect.any(Buffer) - ); -}); - - test('should successfully deidentify a generic file and poll until SUCCESS', async () => { - const file = new File(['generic content'], 'test.abc', { type: 'application/octet-stream' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyFile.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'genRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-222') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'genProcessedFile', - processedFileType: 'abc', - processedFileExtension: 'abc', - }, - ], - wordCharacterCount: { - wordCount: 4, - characterCount: 40, - }, - size: 256, - duration: 0, - pages: 0, - slides: 0, - run_id: 'genRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(result.fileBase64).toBe('genProcessedFile'); - expect(result.type).toBe('abc'); - expect(result.extension).toBe('abc'); - expect(result.wordCount).toBe(4); - expect(result.charCount).toBe(40); - expect(result.sizeInKb).toBe(256); - expect(result.status).toBe('SUCCESS'); - }); - - test('should successfully deidentify a PDF file and save processed file to output directory', async () => { - // 1. Clear any previous mock call data - jest.clearAllMocks(); - - const pdfFile = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const pdfRequest = new DeidentifyFileRequest({file: pdfFile}); - - const mockOptions = new DeidentifyFileOptions(); - mockOptions.setWaitTime(16); - mockOptions.setOutputDirectory('/mock/output/directory'); - - // Mock API implementations... - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'mockRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-123') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ - processedFile: Buffer.from('mockProcessedFileContent').toString('base64'), - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }], - // ... (other mock data) - }); - - // 2. Setup Spies (source uses fs.promises — spy on async methods) - const mkdirSpy = jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - const writeSpy = jest.spyOn(fs.promises, 'writeFile').mockResolvedValue(undefined); - - // 3. START the execution - const promise = detectController.deidentifyFile(pdfRequest, mockOptions); - - // 4. ADVANCE timers so the polling intervals fire - await jest.runAllTimersAsync(); - - // 5. AWAIT the result - const result = await promise; - - // Assertions - expect(result.extension).toBe('pdf'); - expect(mkdirSpy).toHaveBeenCalledWith('/mock/output/directory', { recursive: true }); - expect(writeSpy).toHaveBeenCalledWith( - expect.stringContaining('processed-test.pdf'), - expect.any(Buffer) - ); -}); - - // Lines 101-119, 319, 621-631: TEXT file type via .text extension - test('should successfully deidentify a text file using deidentifyText API', async () => { - jest.clearAllMocks(); - const file = new File(['text content'], 'test.text', { type: 'text/plain' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyText.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'textRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('request-id-text') } }, - }), - })); - - mockVaultClient.filesAPI.getRun - .mockResolvedValueOnce({ status: 'IN_PROGRESS' }) - .mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ - processedFile: 'textProcessedFile', - processedFileType: 'text', - processedFileExtension: 'txt', - }], - wordCharacterCount: { wordCount: 3, characterCount: 30 }, - size: 128, - duration: 0, - pages: 0, - slides: 0, - run_id: 'textRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(mockVaultClient.filesAPI.deidentifyText).toHaveBeenCalled(); - expect(result.fileBase64).toBe('textProcessedFile'); - expect(result.status).toBe('SUCCESS'); - }); - - // Line 483: entities array populated from output with processedFileType === 'entities' - test('should populate entities array when output contains entities type items', async () => { - jest.clearAllMocks(); - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'entRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-ent') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { - processedFile: 'mainProcessedFile', - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }, - { - processedFile: 'entitiesData', - processedFileType: 'entities', - processedFileExtension: 'json', - }, - ], - wordCharacterCount: { wordCount: 5, characterCount: 50 }, - size: 512, - duration: 0, - pages: 1, - slides: 0, - run_id: 'entRunId', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(result.entities).toHaveLength(1); - expect(result.entities[0].file).toBe('entitiesData'); - expect(result.entities[0].extension).toBe('json'); - }); - - // Lines 347-348 + 720: maxWaitTime=1 reached on first poll → resolve IN_PROGRESS → early return - test('should return IN_PROGRESS response when maxWaitTime reached during first poll', async () => { - jest.clearAllMocks(); - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setWaitTime(1); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'pollRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-poll') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValue({ status: 'IN_PROGRESS' }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(result.runId).toBe('pollRunId'); - expect(result.status).toBe('IN_PROGRESS'); - }); - - // Lines 353-354: nextWaitTime >= maxWaitTime → cap currentWaitTime then hit max on second poll - test('should cap wait time when nextWaitTime exceeds maxWaitTime', async () => { - jest.clearAllMocks(); - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setWaitTime(2); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'capRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-cap') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValue({ status: 'IN_PROGRESS' }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - const result = await promise; - expect(result.runId).toBe('capRunId'); - expect(result.status).toBe('IN_PROGRESS'); - }); - - // Lines 366-367: FAILED status via correct {file} constructor - test('should reject when polling returns FAILED status (correct request wrapper)', async () => { - jest.clearAllMocks(); - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'failRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-fail') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValue({ status: 'FAILED', message: 'Processing failed' }); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow(); - }); - - // Lines 368-369: unknown status via correct {file} constructor - test('should reject when polling returns unknown status', async () => { - jest.clearAllMocks(); - const file = new File(['dummy content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'unknownRunId' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-unknown') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValue({ status: 'UNKNOWN_STATUS', message: 'unexpected' }); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow(); - }); - - // Line 285: null processedFile → continue (skip writing) - test('should skip output item when processedFile is null', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/mock/output'); - - const mkdirSpy = jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - const writeSpy = jest.spyOn(fs.promises, 'writeFile').mockResolvedValue(undefined); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runNull' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-null') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { processedFile: null, processedFileExtension: 'pdf' }, - ], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 0, duration: 0, pages: 0, slides: 0, - run_id: 'runNull', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - await promise; - expect(mkdirSpy).toHaveBeenCalled(); - expect(writeSpy).not.toHaveBeenCalled(); - }); - - // Line 289: non-alphanumeric extension → throw SkyflowError - test('should throw when processedFileExtension contains non-alphanumeric characters', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/mock/output'); - - jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runBadExt' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-badext') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { processedFile: 'base64data', processedFileExtension: '../etc' }, - ], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 0, duration: 0, pages: 0, slides: 0, - run_id: 'runBadExt', - }); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow(); - }); - - // Lines 301-302: json extension → Buffer.from().toString(utf8) → writeFile with string - test('should write UTF-8 decoded content for json extension', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/mock/output'); - - const mkdirSpy = jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - const writeSpy = jest.spyOn(fs.promises, 'writeFile').mockResolvedValue(undefined); - - const base64Json = Buffer.from('{"key":"value"}').toString('base64'); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runJson' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-json') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { processedFile: base64Json, processedFileType: 'json', processedFileExtension: 'json' }, - ], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 0, duration: 0, pages: 0, slides: 0, - run_id: 'runJson', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - await promise; - expect(writeSpy).toHaveBeenCalledWith( - expect.stringContaining('processed-test.json'), - '{"key":"value"}' - ); - }); - - // Lines 303-305: mp3 extension → Buffer.from() → writeFile with binary encoding - test('should write binary content for mp3 extension', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/mock/output'); - - const mkdirSpy = jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - const writeSpy = jest.spyOn(fs.promises, 'writeFile').mockResolvedValue(undefined); - - const base64Mp3 = Buffer.from('mp3 audio data').toString('base64'); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runMp3' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-mp3') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { processedFile: base64Mp3, processedFileType: 'mp3', processedFileExtension: 'mp3' }, - ], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 0, duration: 0, pages: 0, slides: 0, - run_id: 'runMp3', - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - - await promise; - expect(writeSpy).toHaveBeenCalledWith( - expect.stringContaining('processed-test.mp3'), - expect.any(Buffer), - { encoding: 'binary' } - ); - }); - - // Line 297: path traversal detected → throw SkyflowError - test('should throw when resolved output path escapes the output directory', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/mock/output'); - - const pathModule = require('path'); - const resolveSpy = jest.spyOn(pathModule, 'resolve') - .mockReturnValueOnce('/etc/malicious.pdf') - .mockReturnValueOnce('/mock/output'); - - jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runTraversal' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-traversal') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { processedFile: 'base64data', processedFileType: 'pdf', processedFileExtension: 'pdf' }, - ], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 0, duration: 0, pages: 0, slides: 0, - run_id: 'runTraversal', - }); - - try { - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow(); - } finally { - resolveSpy.mockRestore(); +describe('skyvault DetectController', () => { + test('extends the shared BaseDetectController and is built from a VaultClient', () => { + const client = new VaultClient('https://abc.vault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + const controller = new DetectController(client); + expect(controller).toBeInstanceOf(BaseDetectController); + for (const method of ['deidentifyString', 'deidentifyFile', 'reidentifyString', 'reidentifyFile', 'getRun', 'checkGuardrails']) { + expect(typeof controller[method]).toBe('function'); } }); - // Line 274: writeFile throws inside decodeBase64AndSaveToFile → SkyflowError - test('should throw SkyflowError when writeFile fails in decodeBase64AndSaveToFile', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - options.setOutputDirectory('/mock/output'); - - jest.spyOn(fs.promises, 'mkdir').mockResolvedValue(undefined); - jest.spyOn(fs.promises, 'writeFile').mockRejectedValue(new Error('Disk full')); - - const base64Data = Buffer.from('some content').toString('base64'); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runWrite' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-write') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [ - { processedFile: base64Data, processedFileType: 'bin', processedFileExtension: 'bin' }, - ], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 0, duration: 0, pages: 0, slides: 0, - run_id: 'runWrite', - }); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow(); - }); - - // Lines 82-99: buildAudioRequest options?.xxx false branches (options undefined) - test('should deidentify audio file without options', async () => { - jest.clearAllMocks(); - const file = new File(['audio content'], 'test.mp3'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyAudio.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'audioNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-audio-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'audioData', processedFileType: 'mp3', processedFileExtension: 'mp3' }], - wordCharacterCount: { wordCount: 1, characterCount: 10 }, - size: 100, duration: 5, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - }); - - // Lines 100-120: buildTextFileRequest options?.xxx false branches (options undefined) - test('should deidentify text file without options (covers buildTextFileRequest options?.xxx false branches)', async () => { - jest.clearAllMocks(); - // .text extension (contains 'text') routes to TEXT handler / buildTextFileRequest - const file = new File(['text content'], 'test.text'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyText.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'textNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-text-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'textData', processedFileType: 'text', processedFileExtension: 'text' }], - wordCharacterCount: { wordCount: 1, characterCount: 10 }, - size: 100, duration: 0, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - }); - - // Lines 129-138: buildPdfRequest options?.xxx false branches (options undefined) - test('should deidentify pdf file without options', async () => { - jest.clearAllMocks(); - const file = new File(['pdf content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'pdfNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-pdf-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'pdfData', processedFileType: 'pdf', processedFileExtension: 'pdf' }], - wordCharacterCount: { wordCount: 5, characterCount: 50 }, - size: 512, duration: 0, pages: 1, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - expect(result.type).toBe('pdf'); - }); - - // Lines 151-161: buildImageRequest options?.xxx false branches (options undefined) - test('should deidentify image file without options', async () => { - jest.clearAllMocks(); - const file = new File(['image content'], 'test.png', { type: 'image/png' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyImage.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'imgNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-img-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'imgData', processedFileType: 'png', processedFileExtension: 'png' }], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 256, duration: 0, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - }); - - // Lines 175-182: buildPptRequest options?.xxx false branches (options undefined) - test('should deidentify pptx file without options', async () => { - jest.clearAllMocks(); - const file = new File(['ppt content'], 'test.pptx'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyPresentation.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'pptNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-ppt-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'pptData', processedFileType: 'pptx', processedFileExtension: 'pptx' }], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 128, duration: 0, pages: 0, slides: 5, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - }); - - // Lines 195-202: buildSpreadsheetRequest options?.xxx false branches (options undefined) - test('should deidentify xlsx file without options', async () => { - jest.clearAllMocks(); - const file = new File(['sheet content'], 'test.xlsx'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifySpreadsheet.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'xlsxNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-xlsx-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'xlsxData', processedFileType: 'xlsx', processedFileExtension: 'xlsx' }], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 128, duration: 0, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - }); - - // Lines 215-223: buildStructuredTextRequest options?.xxx false branches (options undefined) - test('should deidentify json file without options', async () => { - jest.clearAllMocks(); - const file = new File(['{"key":"val"}'], 'test.json'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyStructuredText.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'jsonNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-json-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'jsonData', processedFileType: 'json', processedFileExtension: 'json' }], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 64, duration: 0, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - }); - - // Lines 236-243: buildDocumentRequest options?.xxx false branches (options undefined) - test('should deidentify docx file without options', async () => { - jest.clearAllMocks(); - const file = new File(['doc content'], 'test.docx'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyDocument.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'docxNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-docx-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'docxData', processedFileType: 'docx', processedFileExtension: 'docx' }], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 64, duration: 0, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); + test('VaultClient.resolveAuthToken resolves through getBearerToken and caches the auth info', async () => { + getBearerToken.mockResolvedValue({ key: 'tok-abc', type: AuthType.API_KEY }); + const client = new VaultClient('https://abc.vault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + await expect(client.resolveAuthToken()).resolves.toBe('tok-abc'); + expect(getBearerToken).toHaveBeenCalledWith({ apiKey: 'sky-key' }, LogLevel.ERROR); + expect(client.authInfo).toEqual({ key: 'tok-abc', type: AuthType.API_KEY }); + // The cached API key is reused on the next resolution. + expect(client.getCredentials()).toEqual({ apiKey: 'tok-abc' }); }); - // Lines 257-264: buildGenericFileRequest options?.xxx false branches (options undefined) - test('should deidentify generic file without options', async () => { - jest.clearAllMocks(); - const file = new File(['generic content'], 'test.abc'); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - - mockVaultClient.filesAPI.deidentifyFile.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'abcNoOpts' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-abc-no') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ processedFile: 'abcData', processedFileType: 'abc', processedFileExtension: 'abc' }], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 64, duration: 0, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); + test('ships the skyvault source enums: SKYFLOW_ID for de-identification only', () => { + expect(Object.values(DeidentifySourceType)).toEqual(['BASE64', 'SKYFLOW_ID', 'PRESIGNED_URL']); + expect(Object.values(ReidentifySourceType)).toEqual(['BASE64', 'PRESIGNED_URL']); + expect(ReidentifySourceType.SKYFLOW_ID).toBeUndefined(); }); - // Lines 344, 363-366: response.status?.toUpperCase() false branch (status undefined) - test('should reject when getRun returns response with no status field', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runNoStatus' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-no-status') } }, - }), - })); - - // No status field — covers response.status?.toUpperCase() false branch - mockVaultClient.filesAPI.getRun.mockResolvedValue({}); - - await expect(detectController.deidentifyFile(deidentifyFileReq, options)).rejects.toThrow(); - }); - - // Lines 459-460, 473, 481: parseDeidentifyFileResponse with null output array - test('should handle SUCCESS response with null output array', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runNullOut' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-null-out') } }, - }), - })); - - // null output covers data.output?.[0]?.xxx false branches and data.output || [] true branch - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: null, - wordCharacterCount: null, - size: null, - duration: null, - pages: null, - slides: null, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - expect(result.fileBase64).toBe(''); - expect(result.entities).toHaveLength(0); - }); - - // Lines 459-460, 473: parseDeidentifyFileResponse with empty output array - test('should handle SUCCESS response with empty output array', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({file}); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'runEmptyOut' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-empty-out') } }, - }), - })); - - // empty output covers data.output[0] undefined (?.processedFile false branch) - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [], - wordCharacterCount: { wordCount: 0, characterCount: 0 }, - size: 0, duration: 0, pages: 0, slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - const result = await promise; - expect(result.status).toBe('SUCCESS'); - expect(result.fileBase64).toBe(''); - expect(result.entities).toHaveLength(0); + test('resolves run and re-identification output types onto the skyvault enums, unknown values to null', () => { + const client = new VaultClient('https://abc.vault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + const controller = new DetectController(client); + expect(controller.toDeidentifySource('SKYFLOW_ID')).toBe(DeidentifySourceType.SKYFLOW_ID); + expect(controller.toDeidentifySource('BASE64')).toBe(DeidentifySourceType.BASE64); + expect(controller.toDeidentifySource('FILE')).toBeNull(); + expect(controller.toDeidentifySource(null)).toBeNull(); + expect(controller.toReidentifySource('PRESIGNED_URL')).toBe(ReidentifySourceType.PRESIGNED_URL); + expect(controller.toReidentifySource('SKYFLOW_ID')).toBeNull(); }); - const fullOptionsFileCases = [ - { - name: 'audio', - fileName: 'test.mp3', - api: 'deidentifyAudio', - assertRequest: (req) => { - expect(req.token_type.default).toBe('entity_unique_counter'); - expect(req.bleep_gain).toBe(0.5); - expect(req.bleep_frequency).toBe(440); - expect(req.output_transcription).toBe('transcription'); - expect(req.output_processed_audio).toBe(true); - }, - }, - { - name: 'text', - fileName: 'test.text', - api: 'deidentifyText', - assertRequest: (req) => { - expect(req.token_type.entity_only).toEqual(['EMAIL']); - expect(req.transformations.shift_dates.max_days).toBe(10); - }, - }, - { - name: 'image', - fileName: 'test.png', - api: 'deidentifyImage', - assertRequest: (req) => { - expect(req.masking_method).toBe('blackbox'); - expect(req.output_ocr_text).toBe(true); - expect(req.output_processed_image).toBe(true); - }, - }, - { - name: 'ppt', - fileName: 'test.pptx', - api: 'deidentifyPresentation', - assertRequest: (req) => { - expect(req.allow_regex).toEqual(['allow-regex']); - expect(req.restrict_regex).toEqual(['restrict-regex']); - }, - }, - { - name: 'spreadsheet', - fileName: 'test.xlsx', - api: 'deidentifySpreadsheet', - assertRequest: (req) => { - expect(req.entity_types).toEqual(['NAME', 'EMAIL']); - }, - }, - { - name: 'structured text', - fileName: 'test.json', - api: 'deidentifyStructuredText', - assertRequest: (req) => { - expect(req.transformations.shift_dates.min_days).toBe(2); - }, - }, - { - name: 'document', - fileName: 'test.docx', - api: 'deidentifyDocument', - assertRequest: (req) => { - expect(req.token_type.entity_unq_counter).toEqual(['NAME']); - }, - }, - { - name: 'generic file', - fileName: 'test.abc', - api: 'deidentifyFile', - assertRequest: (req) => { - expect(req.entity_types).toEqual(['NAME', 'EMAIL']); - expect(req.transformations.shift_dates.entity_types).toEqual(['DATE']); - }, - }, - ]; + test('getRun and reidentifyFile surface the resolved output type', async () => { + const client = new VaultClient('https://abc.vault.skyflowapis.com', 'vault123', { apiKey: 'sky-key' }, undefined, LogLevel.ERROR); + client.resolveAuthToken = jest.fn().mockResolvedValue('tok'); + const controller = new DetectController(client); + const files = { getRunV2: jest.fn(), reidentifyFileV2: jest.fn() }; + controller.files = () => files; - test.each(fullOptionsFileCases)( - 'should pass all file options to %s deidentify API', - async ({ fileName, api, assertRequest }) => { - jest.clearAllMocks(); - const file = new File(['content'], fileName); - const deidentifyFileReq = new DeidentifyFileRequest({ file }); - const options = createFullDeidentifyFileOptions(); + files.getRunV2.mockResolvedValue({ status: 'SUCCESS', outputType: 'SKYFLOW_ID', output: [] }); + expect((await controller.getRun(new GetRunRequest('run-1'))).outputType).toBe(DeidentifySourceType.SKYFLOW_ID); - mockVaultClient.filesAPI[api].mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: `run-${api}` }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ - processedFile: 'data', - processedFileType: 'txt', - processedFileExtension: 'txt', - }], - wordCharacterCount: { wordCount: 1, characterCount: 10 }, - size: 10, - duration: 0, - pages: 0, - slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - await promise; - - expect(mockVaultClient.filesAPI[api]).toHaveBeenCalled(); - assertRequest(mockVaultClient.filesAPI[api].mock.calls[0][0]); - } - ); - - test('should poll when deidentify response has null data', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({ file }); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: null, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, + files.reidentifyFileV2.mockResolvedValue({ status: 'SUCCESS', outputType: 'PRESIGNED_URL', output: [] }); + const reid = await controller.reidentifyFile( + new ReidentifyFileRequest(ReidentifySourceType.PRESIGNED_URL, 'https://u', { + redactionLevel: [{ entityName: EntityType.NAME, redactionType: RedactionType.MASKED }], }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ - processedFile: 'pdfData', - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }], - wordCharacterCount: { wordCount: 1, characterCount: 10 }, - size: 10, - duration: 0, - pages: 1, - slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - await promise; - - expect(mockVaultClient.filesAPI.getRun).toHaveBeenCalledWith( - undefined, - { vault_id: 'vault123' } ); + expect(reid.outputType).toBe(ReidentifySourceType.PRESIGNED_URL); + expect(files.reidentifyFileV2.mock.calls[0][0].redactionLevel).toEqual([{ entityName: 'NAME', redactionType: 'MASKED' }]); }); - - test('should poll when deidentify response omits run_id', async () => { - jest.clearAllMocks(); - const file = new File(['content'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({ file }); - const options = new DeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: {}, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ - processedFile: 'pdfData', - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }], - wordCharacterCount: { wordCount: 1, characterCount: 10 }, - size: 10, - duration: 0, - pages: 1, - slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - await promise; - - expect(mockVaultClient.filesAPI.getRun).toHaveBeenCalledWith( - undefined, - { vault_id: 'vault123' } - ); - }); - - test('should pass all file options to pdf deidentify API', async () => { - jest.clearAllMocks(); - const file = new File(['pdf'], 'test.pdf', { type: 'application/pdf' }); - const deidentifyFileReq = new DeidentifyFileRequest({ file }); - const options = createFullDeidentifyFileOptions(); - - mockVaultClient.filesAPI.deidentifyPdf.mockImplementation(() => ({ - withRawResponse: jest.fn().mockResolvedValue({ - data: { run_id: 'run-pdf' }, - rawResponse: { headers: { get: jest.fn().mockReturnValue('req-id') } }, - }), - })); - - mockVaultClient.filesAPI.getRun.mockResolvedValueOnce({ - status: 'SUCCESS', - output: [{ - processedFile: 'pdfData', - processedFileType: 'pdf', - processedFileExtension: 'pdf', - }], - wordCharacterCount: { wordCount: 1, characterCount: 10 }, - size: 10, - duration: 0, - pages: 1, - slides: 0, - }); - - const promise = detectController.deidentifyFile(deidentifyFileReq, options); - await jest.runAllTimersAsync(); - await promise; - - const pdfReq = mockVaultClient.filesAPI.deidentifyPdf.mock.calls[0][0]; - expect(pdfReq.max_resolution).toBe(1920); - expect(pdfReq.density).toBe(300); - expect(pdfReq.token_type.default).toBe('entity_unique_counter'); - }); -}); \ No newline at end of file +}); diff --git a/skyvault/test/vault/model/deprecated.test.js b/skyvault/test/vault/model/deprecated.test.js index 1169c62b..ab37e7ab 100644 --- a/skyvault/test/vault/model/deprecated.test.js +++ b/skyvault/test/vault/model/deprecated.test.js @@ -10,10 +10,7 @@ import { printLog, OrderByEnum, RedactionType } from "skyflow-common"; import DetokenizeOptions from "../../../src/vault/model/options/detokenize"; import GetOptions from "../../../src/vault/model/options/get"; import FileUploadRequest from "../../../src/vault/model/request/file-upload"; -import { Bleep } from "../../../src/vault/model/options/deidentify-file/bleep-audio"; import FileUploadOptions from "../../../src/vault/model/options/fileUpload"; -import DeidentifyTextResponse from "../../../src/vault/model/response/deidentify-text"; -import DeidentifyFileResponse from "../../../src/vault/model/response/deidentify-file"; beforeEach(() => { printLog.mockClear(); @@ -156,42 +153,6 @@ describe("GetOptions deprecated methods", () => { }); }); -describe("Bleep", () => { - test("setStartPadding stores value retrieved by getStartPadding", () => { - const b = new Bleep(); - b.setStartPadding(0.5); - expect(b.getStartPadding()).toBe(0.5); - }); - - test("getStartPadding returns undefined when not set", () => { - const b = new Bleep(); - expect(b.getStartPadding()).toBeUndefined(); - }); - - test("setStopPadding stores value retrieved by getStopPadding", () => { - const b = new Bleep(); - b.setStopPadding(1.2); - expect(b.getStopPadding()).toBe(1.2); - }); - - test("getStopPadding returns undefined when not set", () => { - const b = new Bleep(); - expect(b.getStopPadding()).toBeUndefined(); - }); - - test("setGain stores value retrieved by getGain", () => { - const b = new Bleep(); - b.setGain(0.8); - expect(b.getGain()).toBe(0.8); - }); - - test("setFrequency stores value retrieved by getFrequency", () => { - const b = new Bleep(); - b.setFrequency(440); - expect(b.getFrequency()).toBe(440); - }); -}); - describe("FileUploadOptions", () => { test("setSkyflowId stores value retrieved by getSkyflowId", () => { const opts = new FileUploadOptions(); @@ -395,56 +356,3 @@ describe("FileUploadRequest full coverage", () => { expect(req.getLegacySkyflowId()).toBe("sky-999"); }); }); - -describe("DeidentifyTextResponse errors branch", () => { - test("errors defaults to null when not provided", () => { - const r = new DeidentifyTextResponse({ - processedText: "text", - entities: [], - wordCount: 1, - charCount: 4, - }); - expect(r.errors).toBeNull(); - }); - - test("errors is set when provided as array", () => { - const err = { requestId: "req-1", description: "fail" }; - const r = new DeidentifyTextResponse({ - processedText: "text", - entities: [], - wordCount: 1, - charCount: 4, - errors: [err], - }); - expect(r.errors).toEqual([err]); - }); - - test("errors is null when explicitly passed null", () => { - const r = new DeidentifyTextResponse({ - processedText: "text", - entities: [], - wordCount: 1, - charCount: 4, - errors: null, - }); - expect(r.errors).toBeNull(); - }); -}); - -describe("DeidentifyFileResponse errors branch", () => { - test("errors defaults to null when not provided", () => { - const r = new DeidentifyFileResponse({}); - expect(r.errors).toBeNull(); - }); - - test("errors is set when provided as array", () => { - const err = { requestId: "req-2", description: "fail" }; - const r = new DeidentifyFileResponse({ errors: [err] }); - expect(r.errors).toEqual([err]); - }); - - test("errors is null when explicitly passed null", () => { - const r = new DeidentifyFileResponse({ errors: null }); - expect(r.errors).toBeNull(); - }); -}); From 509446cc622e02a8f3fff5a2f9aa662d54503a6f Mon Sep 17 00:00:00 2001 From: skyflow-bharti Date: Mon, 9 Nov 2026 09:26:08 +0530 Subject: [PATCH 08/11] SK-3076 add README for each sdk --- README.md | 1414 +---------------- common/src/detect/internal/request-mapper.ts | 24 +- common/src/detect/types.ts | 38 +- common/test/detect/request-mapper.test.js | 69 +- docs/api_reference.md | 17 +- docs/migrate_to_v2.md | 2 +- flowvault/README.md | 6 +- .../api-report/skyflow-flowvault-node.api.md | 27 + skyvault/README.md | 122 +- skyvault/api-report/skyflow-node.api.md | 440 ++--- skyvault/src/index.ts | 2 + 11 files changed, 411 insertions(+), 1750 deletions(-) diff --git a/README.md b/README.md index 55cb491f..3dc3a871 100644 --- a/README.md +++ b/README.md @@ -1,1407 +1,41 @@ -# Skyflow Node.js SDK +# Skyflow Node.js -> **This is the current, recommended version of the Skyflow SDK.** V2.1.0 brings flexible auth, multi-vault support, builder patterns, native data types, and rich error diagnostics. -> -> Migrating from v1? See the **[Migration Guide](docs/migrate_to_v2.md)** for step-by-step instructions. V1 is in maintenance mode and will reach End of Life on October 31, 2026. - -Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow SDK for Node.js, Deno, Bun, and Cloudflare Workers. +This repository hosts Skyflow's Node.js SDKs for integrating Skyflow into a Node.js, Deno, Bun, or Cloudflare Workers backend. It's a single npm workspace with more than one published package — pick the package that matches what you need below. [![CI](https://img.shields.io/static/v1?label=CI&message=passing&color=green?style=plastic&logo=github)](https://github.com/skyflowapi/skyflow-node/actions) [![GitHub release](https://badge.fury.io/js/skyflow-node.svg)](https://www.npmjs.com/package/skyflow-node) [![License](https://img.shields.io/github/license/skyflowapi/skyflow-node)](https://github.com/skyflowapi/skyflow-node/blob/main/LICENSE) -## Table of contents - -- [Skyflow Node.js SDK](#skyflow-nodejs-sdk) - - [Table of contents](#table-of-contents) - - [Overview](#overview) - - [Installation](#installation) - - [Require](#require) - - [ES modules](#es-modules) - - [All imports](#all-imports) - - [Quickstart](#quickstart) - - [Before you begin](#before-you-begin) - - [Authenticate](#authenticate) - - [API Key](#api-key) - - [Bearer Token (static)](#bearer-token-static) - - [Initialize the client](#initialize-the-client) - - [Insert data into the vault, get tokens back](#insert-data-into-the-vault-get-tokens-back) - - [Upgrade from v1 to v2](#upgrade-from-v1-to-v2) - - [Vault](#vault) - - [Insert and tokenize data: `.insert(request)`](#insert-and-tokenize-data-insertrequest) - - [Insert example with `continueOnError` option](#insert-example-with-continueonerror-option) - - [Upsert request](#upsert-request) - - [Detokenize: `.detokenize(request, options)`](#detokenize-detokenizerequest-options) - - [Construct a detokenize request](#construct-a-detokenize-request) - - [Get Record(s): `.get(request, options)`](#get-records-getrequest-options) - - [Construct a get request](#construct-a-get-request) - - [Get by Skyflow IDs](#get-by-skyflow-ids) - - [Get tokens for records](#get-tokens-for-records) - - [Get by column name and column values](#get-by-column-name-and-column-values) - - [Redaction Types](#redaction-types) - - [Update Records](#update-records) - - [Construct an update request](#construct-an-update-request) - - [Delete Records](#delete-records) - - [Query](#query) - - [Upload File](#upload-file) - - [Retrieve Existing Tokens: `.tokenize(request)`](#retrieve-existing-tokens-tokenizerequest) - - [Construct a `.tokenize()` request](#construct-a-tokenize-request) - - [Detect](#detect) - - [De-identify a string: `.deidentifyString(request)`](#de-identify-a-string-deidentifystringrequest) - - [Detect configuration](#detect-configuration) - - [De-identify a file: `.deidentifyFile(request)`](#de-identify-a-file-deidentifyfilerequest) - - [Re-identify a string: `.reidentifyString(request)`](#re-identify-a-string-reidentifystringrequest) - - [Re-identify a file: `.reidentifyFile(request)`](#re-identify-a-file-reidentifyfilerequest) - - [Get run: `.getRun(request)`](#get-run-getrunrequest) - - [Check guardrails: `.checkGuardrails(request)`](#check-guardrails-checkguardrailsrequest) - - [Connections](#connections) - - [Configure a connection](#configure-a-connection) - - [Invoke a connection](#invoke-a-connection) - - [Construct an invoke connection request](#construct-an-invoke-connection-request) - - [Authentication \& authorization](#authentication--authorization) - - [Types of `credentials`](#types-of-credentials) - - [Generate bearer tokens for authentication \& authorization](#generate-bearer-tokens-for-authentication--authorization) - - [Generate a bearer token](#generate-a-bearer-token) - - [`generateBearerToken(filepath)`](#generatebearertokenfilepath) - - [`generateBearerTokenFromCreds(credentialsString)`](#generatebearertokenfromcredscredentialsstring) - - [Generate bearer tokens scoped to certain roles](#generate-bearer-tokens-scoped-to-certain-roles) - - [Generate bearer tokens with `ctx` for context-aware authorization](#generate-bearer-tokens-with-ctx-for-context-aware-authorization) - - [Generate signed data tokens](#generate-signed-data-tokens) - - [Client management](#client-management) - - [Using the client in production](#using-the-client-in-production) - - [Logging](#logging) - - [Example `skyflowConfig.logLevel: LogLevel.INFO`](#example-skyflowconfigloglevel-loglevelinfo) - - [Error handling](#error-handling) - - [Catching `SkyflowError` instances](#catching-skyflowerror-instances) - - [Per-record errors (`SkyflowRecordError`)](#per-record-errors-skyflowrecorderror) - - [Bearer token expiration edge cases](#bearer-token-expiration-edge-cases) - - [Troubleshooting](#troubleshooting) - - [TypeScript types reference](#typescript-types-reference) - - [Security](#security) - - [Reporting a Vulnerability](#reporting-a-vulnerability) - -## Overview - -The Skyflow SDK enables you to connect to your Skyflow Vault(s) to securely handle sensitive data at rest, in-transit, and in-use. - -> [!TIP] -> Looking for the full list of request parameters, options setters, response fields, enums, and Detect helper classes? See the **[API Reference](docs/api_reference.md)**. - -> [!IMPORTANT] -> This readme documents SDK version 2. -> For version 1 see the [v1.14.2 README](https://github.com/skyflowapi/skyflow-node/tree/1.14.2). -> For more information on how to migrate see [MIGRATE_TO_V2.md](docs/migrate_to_v2.md). - -## Installation - -Requires Node v12.22.12 and above. - -```sh -npm install skyflow-node -``` - -Depending on your project setup, you may use either the `require` method (common in Node.js projects) or the `import` statement (common in projects using ES modules). - -### Require - -```typescript -const { Skyflow } = require("skyflow-node"); -``` - -### ES modules - -```typescript -import { Skyflow } from "skyflow-node"; -``` - -### All imports - -```typescript -import { - Skyflow, // Vault client - isExpired, // JWT auth helpers - LogLevel, // logging options -} from "skyflow-node"; -``` - -## Quickstart - -Get started quickly with the essential steps: authenticate, initialize the client, and perform a basic vault operation. This section shows you a minimal working example. - -### Before you begin - -You need a Skyflow account and a few values from Skyflow Studio. If you don't have an account, [request a demo](https://www.skyflow.com/get-demo). - -| Value | Where to find it | -|---|---| -| `vaultId` | Your vault's details page in Skyflow Studio. | -| `clusterId` | The first segment of your vault URL: `https://{clusterId}.vault.skyflowapis.com`. | -| `env` | The environment your vault runs in — `Env.PROD`, `Env.SANDBOX`, `Env.DEV`, or `Env.STAGE` (defaults to `PROD`). | -| Credentials | Create a **service account** in Studio. Choose **API key** during creation for the simplest setup, or download the `credentials.json` file for token-based auth. See [Authentication & authorization](#authentication--authorization). | - -The quickstart below assumes a table named `table1` with `card_number` and `cardholder_name` columns. Adjust the table and column names to match your vault schema. See the [Skyflow docs](https://docs.skyflow.com/) for creating vaults, tables, and service accounts. - -### Authenticate - -You can use an API key or a personal bearer token to directly authenticate and authorize requests with the SDK. Use API keys for long-term service authentication. Use bearer tokens for optimal security. - -### API Key - -```typescript -import { Credentials } from 'skyflow-node'; - -const credentials: Credentials = { - apiKey: "", -}; -``` - -### Bearer Token (static) - -```typescript -import { Credentials } from 'skyflow-node'; - -const credentials: Credentials = { - token: "", -}; -``` - -For authenticating via generated bearer tokens including support for scoped tokens, context-aware access tokens, and more, refer to the [Authentication & Authorization](#authentication--authorization) section. - -### Initialize the client - -Initialize the Skyflow client first. You can specify different credential types during initialization. - -```javascript -import { Skyflow, SkyflowConfig, VaultConfig, Env, LogLevel } from 'skyflow-node'; - -// Create a credentials object. We'll use an API key. -const skyflowCredentials = { - apiKey: "" -}; - -// Configure vault -const vaultConfig: VaultConfig = { - vaultId: '', // required: vault identifier - clusterId: '', // required: from vault URL - env: Env.PROD, // optional: PROD (default), SANDBOX, DEV, STAGE - // credentials: vaultCredentials // optional: per-vault credentials; overrides skyflowCredentials -}; - -// Initialize Skyflow client -const skyflowConfig: SkyflowConfig = { - vaultConfigs: [vaultConfig], - skyflowCredentials: skyflowCredentials, - logLevel: LogLevel.ERROR -}; - -const skyflowClient: Skyflow = new Skyflow(skyflowConfig); -``` - -See [docs/advanced_initialization.md](docs/advanced_initialization.md) for advanced initialization examples including multiple vaults and different credential types. - -### Insert data into the vault, get tokens back - -Insert data into your vault using the `insert` method. Set `insertOptions.setReturnTokens(true)` to ensure values are tokenized in the response. - -Create an insert request with the [`InsertRequest`](docs/api_reference.md#insertrequest) class, which includes the values to be inserted as a list of records. - -Below is a simple example to get started. See the [Insert and tokenize data](#insert-and-tokenize-data-insertrequest) section for advanced options. - -```javascript -import { InsertRequest, InsertOptions } from "skyflow-node"; - -// Insert sensitive data into the vault -const insertData = [ - { card_number: "4111111111111112", cardholder_name: "John Doe" }, -]; -const insertReq = new InsertRequest("table1", insertData); - -const insertOptions = new InsertOptions(); -insertOptions.setReturnTokens(true); - -const insertResponse = await skyflowClient - .vault(vaultId) - .insert(insertReq, insertOptions); - -console.log("Insert response:", insertResponse); -``` - -Returns an [`InsertResponse`](docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setReturnTokens(true)`, each entry includes the `skyflowId` and a token per column: - -```text -Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '5391-4629-3722-7102', cardholder_name: '0f6b8a2c-90ab-4cde-9def-567890abcdef' }], errors: null } -``` - -## Upgrade from v1 to v2 - -Upgrade from `skyflow-node` v1 using the dedicated guide in [docs/migrate_to_v2.md](docs/migrate_to_v2.md). - -## Vault - -The [Vault](https://docs.skyflow.com/docs/vaults) performs operations on the vault such as inserting records, detokenizing tokens, retrieving tokens for list of `skyflowId`s and to invoke the Connection. - -### Insert and tokenize data: `.insert(request)` - -Pass options to the `insert` method to enable additional functionality such as returning tokenized data, upserting records, or allowing bulk operations to continue despite errors. See [Quickstart](#quickstart) for a basic example. - -```typescript -import { InsertRequest, InsertResponse } from 'skyflow-node'; - -const insertRequest = new InsertRequest('table1', [ - { - : '', - : '', - }, - { - : '', - : '', - }, -]); - -const response: InsertResponse = await skyflowClient - .vault('') - .insert(insertRequest); - -console.log('Insert response:', response); -``` - -Returns an [`InsertResponse`](docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setContinueOnError(true)`, `insertedFields` contains successful records and `errors` contains per-record failures: - -```text -Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', '': '', '': '' }], errors: null } -``` - -> **Note:** The response key is `skyflowId`. The legacy `skyflow_id` key is deprecated and will be removed in an upcoming release. - -#### Insert example with `continueOnError` option - -Set the `continueOnError` flag to `true` to allow insert operations to proceed despite encountering partial errors. - -> [!TIP] -> See the full example in the samples directory: [insert-continue-on-error.ts](samples/vault-api/insert-continue-on-error.ts) - -#### Upsert request - -Turn an insert into an 'update-or-insert' operation using the upsert option. The vault checks for an existing record with the same value in the specified column. If a match exists, the record updates; otherwise, a new record inserts. - -```typescript -// Specify the column to use as the index for the upsert. -// Note: The column must have the `unique` constraint configured in the vault. -insertOptions.setUpsertColumn("cardholder_name"); -``` - -For all `InsertOptions` setters and `TokenMode` values, see [InsertOptions](docs/api_reference.md#insertoptions) in the API reference. - -### Detokenize: `.detokenize(request, options)` - -Convert tokens back into plaintext values (or masked values) using the `.detokenize()` method. Detokenization accepts tokens and returns values. - -Create a detokenization request with the [`DetokenizeRequest`](docs/api_reference.md#detokenizerequest) class, which requires a list of tokens and column groups as input. - -Provide optional parameters such as the redaction type and the option to continue on error. - -#### Construct a detokenize request - -```typescript -import { - DetokenizeOptions, - DetokenizeRequest, - DetokenizeResponse, -} from "skyflow-node"; - -const detokenizeRequest = new DetokenizeRequest([ - { token: "token1", redactionType: RedactionType.PLAIN_TEXT }, - { token: "token2", redactionType: RedactionType.PLAIN_TEXT }, -]); - -const detokenizeOptions = new DetokenizeOptions(); -detokenizeOptions.setContinueOnError(true); -detokenizeOptions.setDownloadUrl(false); - -const response: DetokenizeResponse = await skyflowClient - .vault(primaryVaultConfig.vaultId) - .detokenize(detokenizeRequest, detokenizeOptions); - -console.log("Detokenization response:", response); -``` - -Returns a [`DetokenizeResponse`](docs/api_reference.md#detokenizeresponse) (`detokenizedFields`, `errors`): - -```text -Detokenization response: { detokenizedFields: [{ token: '5391-4629-3722-7102', value: '4111111111111112', type: 'STRING' }], errors: null } -``` - -> [!TIP] -> See the full example in the samples directory: [detokenzie-records.ts](samples/vault-api/detokenzie-records.ts) - -### Get Record(s): `.get(request, options)` - -Retrieve data using Skyflow IDs or unique column values with the get method. Create a get request with the [`GetRequest`](docs/api_reference.md#getrequest) class, specifying parameters such as the table name, redaction type, Skyflow IDs, column names, and column values. - -> [!NOTE] -> You can't use both Skyflow IDs and column name/value pairs in the same request. Use the `GetOptions` class to specify whether to return tokens. - -#### Construct a get request - -```typescript -import { GetRequest, GetOptions, GetResponse } from "skyflow-node"; - -const getRequest = new GetRequest("table1", ["", ""]); - -const getOptions = new GetOptions(); -getOptions.setReturnTokens(false); - -const response: GetResponse = await skyflowClient - .vault("") - .get(getRequest, getOptions); - -console.log("Get response:", response); -``` - -Returns a [`GetResponse`](docs/api_reference.md#getresponse) (`data`, `errors`): - -```text -Get response: { data: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '4111111111111112', cardholder_name: 'John Doe' }], errors: null } -``` - -> **Note:** The response key is `skyflowId`. The legacy `skyflow_id` key is deprecated and will be removed in an upcoming release. - -#### Get by Skyflow IDs - -Retrieve specific records using Skyflow IDs. Use this method when you know the exact record IDs. - -```typescript -import { - GetOptions, - GetRequest, - SkyflowError, - GetResponse, - RedactionType, -} from "skyflow-node"; - -// Initialize a list of Skyflow IDs to retrieve records (replace with actual Skyflow IDs) -const getIds: Array = [ - "a581d205-1969-4350-acbe-a2a13eb871a6", - "5ff887c3-b334-4294-9acc-70e78ae5164a", -]; - -// Step 2: Create a GetRequest to retrieve records by Skyflow ID -const getRequest: GetRequest = new GetRequest( - "table1", // Replace with your actual table name - getIds, -); - -// Step 3: Configure Get Options and specify not to return tokens and redaction type -const getOptions: GetOptions = new GetOptions(); -getOptions.setReturnTokens(false); // Optional: Set to false to avoid returning tokens -getOptions.setRedactionType(RedactionType.PLAIN_TEXT); - -// Step 4: Send the request to the Skyflow vault and retrieve the records -const getResponse: GetResponse = await skyflowClient - .vault(primaryVaultConfig.vaultId) - .get(getRequest, getOptions); -// Replace with your actual Skyflow vault ID - -console.log("Data retrieval successful:", getResponse); -``` - -#### Get tokens for records - -Return tokens for records to securely process sensitive data while maintaining data privacy. - -```ts -getOptions.setReturnTokens(true); // Set to `true` to get tokens -``` - -> [!TIP] -> See the full example in the samples directory: [get-records.ts](samples/vault-api/get-records.ts) - -#### Get by column name and column values - -Retrieve records by unique column values when you don't know the Skyflow IDs. Use this method to query data with alternate unique identifiers. - -```ts -const getRequest: GetColumnRequest = new GetColumnRequest( - tableName, - columnName, - columnValues, // Column values of the records to return -); -``` - -> [!TIP] -> See the full example in the samples directory: [get-column-values.ts](samples/vault-api/get-column-values.ts) - -#### Redaction Types - -Use redaction types to control how sensitive data displays when retrieved from the vault. - -**Available Redaction Types** - -- `DEFAULT`: Applies the vault-configured default redaction setting. -- `REDACTED`: Completely removes sensitive data from view. -- `MASKED`: Partially obscures sensitive information. -- `PLAIN_TEXT`: Displays the full, unmasked data. - -**Choosing the Right Redaction Type** - -- Use `REDACTED` for scenarios requiring maximum data protection to prevent exposure of sensitive information. -- Use `MASKED` to provide partial visibility of sensitive data for less critical use cases. -- Use `PLAIN_TEXT` for internal, authorized access where full data visibility is necessary. - -For all `GetOptions` setters and `OrderByEnum` values, see [GetOptions](docs/api_reference.md#getoptions) in the API reference. - -### Update Records - -Update data in your vault using the `update` method. Create an update request with the [`UpdateRequest`](docs/api_reference.md#updaterequest) class, specifying parameters such as the table name and data (as a dictionary). - -Configure update options using the `UpdateOptions` class to control returnTokens, tokens, and tokenMode. When `returnTokens` is `true`, Skyflow returns tokens for the updated records. When `false`, Skyflow returns IDs for the updated records. - -#### Construct an update request - -```typescript -import { UpdateRequest, UpdateOptions, UpdateResponse, TokenMode } from 'skyflow-node'; - -const updateRequest = new UpdateRequest('table1', { - skyflowId: '', - : '', - : '' -}); - -const updateOptions = new UpdateOptions(); -updateOptions.setReturnTokens(true); // return tokens for updated fields - -const response: UpdateResponse = await skyflowClient - .vault('') - .update(updateRequest, updateOptions); - -console.log('Update response:', response); -``` - -Returns an [`UpdateResponse`](docs/api_reference.md#updateresponse) (`updatedField`, `errors`). With the default `setReturnTokens(false)`, `updatedField` contains only `skyflowId`: - -```text -Update response: { updatedField: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c' }, errors: null } -``` - -> **Note:** The response key is `skyflowId`. The legacy `skyflow_id` key is deprecated and will be removed in an upcoming release. - -For all `UpdateOptions` setters and `TokenMode` values, see [UpdateOptions](docs/api_reference.md#updateoptions) in the API reference. - -> [!TIP] -> See the full example in the samples directory: [update-record.ts](samples/vault-api/update-record.ts) - -### Delete Records - -Delete records using Skyflow IDs with the `delete` method. Create a delete request with the [`DeleteRequest`](docs/api_reference.md#deleterequest) class, which accepts a list of Skyflow IDs: - -```typescript -import { DeleteRequest, DeleteResponse } from "skyflow-node"; - -const deleteRequest = new DeleteRequest("table1", [ - "", - "", - "", -]); - -const response: DeleteResponse = await skyflowClient - .vault("") - .delete(deleteRequest); - -console.log("Delete response:", response); -``` - -Returns a [`DeleteResponse`](docs/api_reference.md#deleteresponse) (`deletedIds`, `errors`): - -```text -Delete response: { deletedIds: ['a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c'], errors: null } -``` - -> [!TIP] -> See the full example in the samples directory: [delete-records.ts](samples/vault-api/delete-records.ts) - -### Query - -Retrieve data with SQL queries using the `query` method. Create a query request with the [`QueryRequest`](docs/api_reference.md#queryrequest) class, which takes the `query` parameter as follows: - - - -```typescript -import { QueryRequest, QueryResponse } from "skyflow-node"; - -const queryRequest = new QueryRequest( - "SELECT * FROM table1 WHERE column1 = 'value'", -); - -const response: QueryResponse = await skyflowClient - .vault("") - .query(queryRequest); - -console.log("Query response:", response); -``` - -Returns a [`QueryResponse`](docs/api_reference.md#queryresponse) (`fields`, `errors`). Each record also includes a `tokenizedData` map: - -```text -Query response: { fields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '4111111111111112', tokenizedData: { card_number: '5391-4629-3722-7102' } }], errors: null } -``` - -> [!TIP] -> See the full example in the samples directory: [query-records.ts](samples/vault-api/query-records.ts) - -Refer to [Query your data](https://docs.skyflow.com/query-data/) and [Execute Query](https://docs.skyflow.com/record/#QueryService_ExecuteQuery) for guidelines and restrictions on supported SQL statements, operators, and keywords. - -### Upload File - -Upload files to a Skyflow vault using the `uploadFile` method. Requires Node.js v20 or above. Create a [`FileUploadRequest`](docs/api_reference.md#fileuploadrequest) with the table name and column name, then configure a `FileUploadOptions` with the file source and optionally a Skyflow ID. - -**Attach a file to an existing record:** - -```typescript -import { FileUploadRequest, FileUploadResponse, FileUploadOptions } from "skyflow-node"; - -const uploadReq = new FileUploadRequest("", ""); - -const uploadOptions = new FileUploadOptions(); -uploadOptions.setSkyflowId(""); // attach to this record -uploadOptions.setFilePath("path/to/file.pdf"); - -const response: FileUploadResponse = await skyflowClient - .vault("") - .uploadFile(uploadReq, uploadOptions); - -console.log("File upload:", response); -``` - -**Upload a file and create a new record (omit `setSkyflowId`):** - -```typescript -const uploadReq = new FileUploadRequest("documents", "attachment"); - -const uploadOptions = new FileUploadOptions(); -uploadOptions.setFilePath("path/to/file.pdf"); // no setSkyflowId — creates a new record - -const response: FileUploadResponse = await skyflowClient - .vault("") - .uploadFile(uploadReq, uploadOptions); - -console.log("File upload:", response); -``` - -Both forms return a [`FileUploadResponse`](docs/api_reference.md#fileuploadresponse) (`skyflowId`, `errors`): - -```text -File upload: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', errors: null } -``` - -For all `FileUploadOptions` setters, see [FileUploadOptions](docs/api_reference.md#fileuploadoptions) in the API reference. - -> [!TIP] -> See the full example in the samples directory: [file-upload.ts](samples/vault-api/file-upload.ts) - -### Retrieve Existing Tokens: `.tokenize(request)` - -Retrieve tokens for values that already exist in the vault using the `.tokenize()` method. This method returns existing tokens only and does not generate new tokens. Build the request with the [`TokenizeRequest`](docs/api_reference.md#tokenizerequest) class. - -#### Construct a `.tokenize()` request - -```typescript -import { TokenizeRequest, TokenizeResponse } from "skyflow-node"; - -const tokenizeRequest = new TokenizeRequest([ - { value: "", columnGroup: "" }, - { value: "", columnGroup: "" }, -]); - -const response: TokenizeResponse = await skyflowClient - .vault("") - .tokenize(tokenizeRequest); - -console.log("Tokenization Result:", response); -``` - -Returns a [`TokenizeResponse`](docs/api_reference.md#tokenizeresponse) (`tokens`, `errors`): - -```text -Tokenization Result: { tokens: ['5391-4629-3722-7102', '0f6b8a2c-90ab-4cde-9def-567890abcdef'], errors: null } -``` - -> [!TIP] -> See the full example in the samples directory: [tokenize-records.ts](samples/vault-api/tokenize-records.ts) - -## Detect - -Skyflow Detect finds and de-identifies sensitive data in free text and files, then re-identifies it on demand. The SDK exposes the Detect V2 API through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault and uses the vault's credentials and log level. - -Detect configurations are created in Studio or through the management API. The SDK does not create, update or delete them. - -| Method | Request | Response | Description | -|--------|---------|----------|-------------| -| `deidentifyString(request)` | [`DeidentifyStringRequest`](docs/api_reference.md#deidentifystringrequest) | [`DeidentifyStringResponse`](docs/api_reference.md#deidentifystringresponse) | Detect and replace sensitive data in a string | -| `deidentifyFile(request)` | [`DeidentifyFileRequest`](docs/api_reference.md#deidentifyfilerequest) | [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) | Submit a file for de-identification, optionally polling the run to completion | -| `reidentifyString(request)` | [`ReidentifyStringRequest`](docs/api_reference.md#reidentifystringrequest) | [`ReidentifyStringResponse`](docs/api_reference.md#reidentifystringresponse) | Replace tokens in a string with their original or redacted values | -| `reidentifyFile(request)` | [`ReidentifyFileRequest`](docs/api_reference.md#reidentifyfilerequest) | [`ReidentifyFileResponse`](docs/api_reference.md#reidentifyfileresponse) | Replace tokens in a file with their original or redacted values | -| `getRun(request)` | [`GetRunRequest`](docs/api_reference.md#getrunrequest) | [`GetRunResponse`](docs/api_reference.md#getrunresponse) | Retrieve the status and outputs of a Detect file run | -| `checkGuardrails(request)` | [`CheckGuardrailsRequest`](docs/api_reference.md#checkguardrailsrequest) | [`CheckGuardrailsResponse`](docs/api_reference.md#checkguardrailsresponse) | Screen text for toxicity and denied topics | - -**Before you start** - -- **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy that grants the Detokenization action on the destination table and columns, unless it is a Vault Owner. -- **Configuration.** `configurationId` is the id of a Detect configuration created through the management API. The alternative is an inline `DetectConfiguration` passed as `configuration` (see [Detect configuration](#detect-configuration)). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings. Sending both is rejected by the API with HTTP 400. -- **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the `table.column` named by `destination`. That table and column must already exist in the vault. -- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](docs/api_reference.md#detect-api). - -### De-identify a string: `.deidentifyString(request)` - -`DeidentifyStringRequest` takes the text plus, optionally, `configurationId` (a stored Detect configuration) or `configuration` (an inline `DetectConfiguration`). The SDK fills the vault id into the configuration from the controller. - -```typescript -import { - DeidentifyStringRequest, - DeidentifyStringResponse, - DetectConfiguration, - EntityType, - DeidentificationType, - ReturnEntitiesType, - SkyflowError, -} from 'skyflow-node'; - -// Stored configuration -const stored = new DeidentifyStringRequest( - 'My name is John Doe, and my email is johndoe@acme.com.', - { configurationId: '' }, -); -const response: DeidentifyStringResponse = await skyflowClient.detect().deidentifyString(stored); -console.log(response.processedText); -// My name is [NAME_1], and my email is [EMAIL_ADDRESS_1]. - -// Inline configuration -const configuration: DetectConfiguration = { - detect: { - entities: [ - { - entityType: EntityType.NAME, - deidentificationType: DeidentificationType.VAULT_TOKEN, - destination: '
.', // table.column that stores the value - }, - { entityType: EntityType.ALL, deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER }, - ], - skip: ['Skyflow'], - returnEntities: ReturnEntitiesType.EXCLUDE_SENSITIVE_DATA, - }, -}; -const inline = new DeidentifyStringRequest( - 'My name is John Doe, and my email is johndoe@acme.com.', - { configuration }, -); -const inlineResponse = await skyflowClient.detect().deidentifyString(inline); -for (const entity of inlineResponse.entities) { - console.log(entity.entityType, entity.token, entity.location?.startIndex, entity.location?.endIndex); -} -console.log(inlineResponse.metrics?.wordCount); -``` - -`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. - -> [!TIP] -> See the full example in the samples directory: [deidentify-string.ts](samples/detect-api/deidentify-string.ts) - -### Detect configuration - -`DetectConfiguration` is the inline form of a stored configuration and accepts the same options. `detect` applies to string and file inputs; `fileMapping` and `media` apply to file inputs only. All of these are plain TypeScript interfaces, so write them as object literals. - -| Type | Fields | Notes | -|------|--------|-------| -| `Detect` | `entities`, `objectEntities`, `restrict`, `skip`, `customTypes`, `returnEntities` | `restrict` and `skip` are regexes that are always, respectively never, treated as sensitive. `returnEntities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. | -| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` is used only with `VAULT_TOKEN`. | -| `Transformation` / `ShiftDates` | `shiftDates`; `minDays`, `maxDays` | Shifts detected dates by a random number of days within the range. | -| `CustomType` | `label`, `match` | A regex-defined entity type. `label` is reported as the entity type of its matches. | -| `ObjectEntity` | `entityType`, `deidentificationType` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. | -| `FileMapping` | `source`, `destination`, `outputFileNameSuffix`, `entitiesDestination`, `objectEntitiesDestination` | Where the processed file and the entity files are written. File inputs only. | -| `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. | -| `Audio` / `Bleep` | `outputProcessedAudio`, `outputTranscription`, `bleep`; `startPadding`, `stopPadding`, `frequency`, `gain` | `outputTranscription` is an `OutputTranscriptionType`. `Bleep` is the tone laid over detected speech; paddings are in seconds, frequency in Hz. | -| `Document` / `Pdf` | `pdf`; `processingMode`, `density`, `maxResolution` | `processingMode` is a `PdfProcessingMode` (`NONE`, `OCR` or `TEXT_LAYER`). `density` is the rendering DPI. | -| `Image` | `outputProcessedImage`, `outputOcrText`, `maskingMethod` | `maskingMethod` is a `MaskingMethod` (`NONE`, `BLUR` or `BLACKBOX`). | - -```typescript -import { - DetectConfiguration, - EntityType, - DeidentificationType, - ObjectEntityType, - ObjectDeidentificationType, - MaskingMethod, - PdfProcessingMode, - OutputTranscriptionType, -} from 'skyflow-node'; - -const configuration: DetectConfiguration = { - detect: { - entities: [ - { - entityType: EntityType.DOB, - deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, - transformation: { shiftDates: { minDays: 10, maxDays: 30 } }, - }, - { entityType: EntityType.ALL }, - ], - objectEntities: [{ entityType: ObjectEntityType.FACE, deidentificationType: ObjectDeidentificationType.REDACT }], - customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], - restrict: ['[0-9]{3}-[0-9]{2}-[0-9]{4}'], - }, - media: { - image: { maskingMethod: MaskingMethod.BLUR, outputOcrText: true }, - document: { pdf: { processingMode: PdfProcessingMode.OCR } }, - audio: { - outputTranscription: OutputTranscriptionType.TRANSCRIPTION, - bleep: { startPadding: 0.5, stopPadding: 0.5 }, - }, - }, -}; -``` - -### De-identify a file: `.deidentifyFile(request)` - -File de-identification is asynchronous on the API: the submit call returns a run id and the result is retrieved from the run. `deidentifyFile` exposes both modes. - -- **Submit only** (default): one call, returns `runId`. `status` is `null` and nothing is written to disk. Retrieve the result later with [`getRun`](#get-run-getrunrequest). -- **Submit and poll**: set `pollOptions`. The SDK polls the run with exponential backoff (1, 2, 4, 8, 16, then 32 seconds between lookups) until it is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `waitTime` (default 60 s, max 300) or `maxAttempts` (default 23, max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED` so you can resume with `getRun`. - -The file is supplied in one of three ways: a `dataSource` plus `value` (`BASE64` content, a `SKYFLOW_ID`, or a `PRESIGNED_URL`), a `file` (a web `File`, Node.js 20+), or a `filePath`. A local file is read, base64-encoded, and its `dataFormat` inferred from the extension. Supply exactly one of `value`, `file` or `filePath`. `configurationId` and `configuration` follow the same rules as for strings. - -`dataFormat` accepts every `FileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. - -`outputDirectory` is meaningful only when polling. Setting it without `pollOptions` fails client-side validation before any request is sent. - -```typescript -import { DeidentifyFileRequest, DeidentifyFileResponse, DetectRunStatus } from 'skyflow-node'; - -// Submit only -const submitted: DeidentifyFileResponse = await skyflowClient.detect().deidentifyFile( - new DeidentifyFileRequest({ - filePath: '/data/intake/patient-notes.pdf', // sent as BASE64, format inferred as PDF - configurationId: '', - }), -); -const runId = submitted.runId; - -// Submit and poll -const response = await skyflowClient.detect().deidentifyFile( - new DeidentifyFileRequest({ - filePath: '/data/intake/patient-notes.pdf', - configurationId: '', - pollOptions: { waitTime: 120 }, - outputDirectory: '/data/processed', // optional - }), -); - -switch (response.status) { - case DetectRunStatus.SUCCESS: - // For local-file requests the outputs are already on disk: - // /data/processed/processed-patient-notes.pdf the redacted file - // /data/processed/processed-entities-patient-notes.json the detected entities - // response.output and response.metrics carry the same data. - break; - case DetectRunStatus.FAILED: - console.error(response.message); - break; - default: // IN_PROGRESS or QUEUED: budget ran out, resume with getRun(runId) - break; -} -``` - -Outputs are written only when polling reached `SUCCESS`, the request was built from a local file, and the run delivered them as base64. A request built from a `SKYFLOW_ID` or `PRESIGNED_URL` has no local name to derive from, and presigned-URL outputs are not fetched, so in those cases read `output` and write the files yourself. Files are named `processed-.`, with the entities output as `processed-entities-.json`; `outputDirectory` is created if missing and defaults to the working directory. Writing outputs needs the Node.js file system and is not supported in Cloudflare Workers. - -> [!TIP] -> See the full examples in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](samples/detect-api/deidentify-file-async.ts) - -### Re-identify a string: `.reidentifyString(request)` - -Re-identification looks tokens up in the vault, so it applies to values that were de-identified with `VAULT_TOKEN`. Placeholders produced by `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not stored anywhere and cannot be re-identified. - -`ReidentifyStringRequest` takes the tokenized text and an optional list of `RedactionLevel` rules. For skyvault vaults each rule names an `entityName` and the `redactionType` to render it with. Without rules every token is replaced by its original value. - -```typescript -import { ReidentifyStringRequest, ReidentifyStringResponse, EntityType, RedactionType } from 'skyflow-node'; - -const response: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( - new ReidentifyStringRequest('My name is [NAME_1].', { - redactionLevel: [{ entityName: EntityType.NAME, redactionType: RedactionType.MASKED }], - }), -); -console.log(response.processedText); -// My name is J**n D*e. -``` - -`RedactionLevel` is the skyvault SDK's own type and has only `entityName` and `redactionType`; the `tokenGroupName` and `redactionPattern` rules of the flowvault SDK do not exist here. Both fields are optional on the API, so the SDK passes the list through as given and the server rejects invalid combinations. - -> [!TIP] -> See the full example in the samples directory: [reidentify-string.ts](samples/detect-api/reidentify-string.ts) - -### Re-identify a file: `.reidentifyFile(request)` - -`reidentifyFile` is synchronous: the re-identified file comes back in the response, not through a run id. `ReidentifyFileRequest` takes a `dataSource` (`ReidentifySourceType.BASE64` or `PRESIGNED_URL`), the matching `value`, an optional `dataFormat` and the same optional `RedactionLevel` rules as `reidentifyString`. `dataFormat` is limited to the formats that support `VAULT_TOKEN`: `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. - -```typescript -import * as fs from 'fs'; -import { ReidentifyFileRequest, ReidentifySourceType, FileDataFormat, DetectRunStatus } from 'skyflow-node'; - -const content = fs.readFileSync('tokenized.txt').toString('base64'); - -const response = await skyflowClient.detect().reidentifyFile( - new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: FileDataFormat.TXT }), -); - -if (response.status === DetectRunStatus.SUCCESS) { - for (const output of response.output) { - // output.processedFile is base64 or a presigned URL, per response.outputType - const processed = Buffer.from(output.processedFile ?? '', 'base64'); - } -} -``` - -`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. - -> [!TIP] -> See the full example in the samples directory: [reidentify-file.ts](samples/detect-api/reidentify-file.ts) - -### Get run: `.getRun(request)` - -File de-identification is asynchronous: the submit call returns a run id, and `getRun` retrieves the run's status and, once it has succeeded, its outputs. `output` is empty and `outputType` and `metrics` are `null` until the run completes. - -```typescript -import { GetRunRequest, GetRunResponse, DetectRunStatus } from 'skyflow-node'; - -const run: GetRunResponse = await skyflowClient.detect().getRun(new GetRunRequest('')); - -switch (run.status) { - case DetectRunStatus.SUCCESS: - for (const output of run.output) { - // output.processedFile is base64 or a presigned URL, per run.outputType - // output.processedFileType is e.g. REDACTED_FILE or ENTITIES - // output.processedFileExtension is a FileDataFormat such as PDF - } - break; - case DetectRunStatus.FAILED: - console.error(run.message); - break; - default: // QUEUED, IN_PROGRESS, UNKNOWN: poll again later - break; -} -``` - -> [!TIP] -> See the full example in the samples directory: [get-run.ts](samples/detect-api/get-run.ts) - -### Check guardrails: `.checkGuardrails(request)` - -`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. - -```typescript -import { CheckGuardrailsRequest, GuardrailsValidation } from 'skyflow-node'; - -const response = await skyflowClient.detect().checkGuardrails( - new CheckGuardrailsRequest(userPrompt, { checkToxicity: true, denyTopics: ['politics', 'medical advice'] }), -); - -if (response.validation === GuardrailsValidation.FAILED) { - throw new Error(`blocked: toxic=${response.toxic} deniedTopic=${response.deniedTopic}`); -} -``` - -All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling). Client-side validation fails before any network call. - -> [!TIP] -> See the full example in the samples directory: [check-guardrails.ts](samples/detect-api/check-guardrails.ts) - -## Connections - -Securely send and receive data between your systems and first- or third-party services using Skyflow Connections, a gateway service that uses tokenization. The [connections](https://github.com/skyflowapi/skyflow-node/tree/v2/src/vault/controller/connections) module invokes both inbound and outbound connections. - -- **Inbound connections**: Act as intermediaries between your client and server, tokenizing sensitive data before it reaches your backend, ensuring downstream services handle only tokenized data. -- **Outbound connections**: Enable secure extraction of data from the vault and transfer it to third-party services via your backend server, such as processing checkout or card issuance flows. - -### Configure a connection - -Add a `ConnectionConfig` to `SkyflowConfig.connectionConfigs` during initialization, or call `addConnectionConfig()` at runtime: - -```typescript -import { ConnectionConfig, Credentials } from 'skyflow-node'; - -const connectionConfig: ConnectionConfig = { - connectionId: '', // required: unique connection identifier - connectionUrl: '', // required: gateway URL for this connection - // credentials: connectionCredentials // optional: per-connection credentials; overrides skyflowCredentials -}; -``` - -### Invoke a connection - -Invoke a connection using the `invoke` method of the Skyflow client. Build the request with the [`InvokeConnectionRequest`](docs/api_reference.md#invokeconnectionrequest) class. - -#### Construct an invoke connection request - -```typescript -import { - InvokeConnectionRequest, - RequestMethod, - InvokeConnectionResponse, -} from "skyflow-node"; - -const invokeRequest = new InvokeConnectionRequest( - RequestMethod.POST, - { : "" }, - { : "" }, - { : "" }, - { : "" } -); - -const response: InvokeConnectionResponse = await skyflowClient - .connection() - .invoke(invokeRequest); - -console.log("Invoke connection response:", response); -``` - -Returns an [`InvokeConnectionResponse`](docs/api_reference.md#invokeconnectionresponse) (`data`, `metadata`, `errors`): - -```text -Invoke connection response: { data: { status: 'success', transaction_id: 'txn_abc123' }, metadata: { requestId: 'req-xyz-456' }, errors: null } -``` - -`method` supports the following methods (see [`RequestMethod`](docs/api_reference.md#requestmethod)): - -- `GET` -- `POST` -- `PUT` -- `PATCH` - -**pathParams, queryParams, header, body** are the JSON objects represented as dictionaries that will be sent through the connection integration url. - -> [!TIP] -> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/vault-api/invoke-connection.ts) -> See [docs.skyflow.com](https://docs.skyflow.com) for more details on integrations with Connections, Functions, and Pipelines. - -## Authentication & authorization - -### Types of `credentials` - -The SDK accepts one of several types of credentials object. - -1. **API keys** - A unique identifier used to authenticate and authorize requests to an API. Use for long-term service authentication. To create an API key, first create a 'Service Account' in Skyflow and choose the 'API key' option during creation. - - ```ts - const credentials: Credentials = { - apiKey: "" - }; - ``` - -2. **Bearer tokens** - A temporary access token used to authenticate API requests. Use for optimal security. As a developer with the right access, you can generate a temporary personal bearer token in Skyflow in the user menu. - - ```ts - const credentials: Credentials = { - token: "" - }; - ``` - -3. **Service account credentials file path** - The file path pointing to a JSON file containing credentials for a service account. Use when credentials are managed externally or stored in secure file systems. - - ```ts - const credentials: Credentials = { - path: "" - }; - ``` - -4. **Service account credentials string** - JSON-formatted string containing service account credentials. Use when integrating with secret management systems or when credentials are passed programmatically. - - ```ts - const credentials: Credentials = { - credentialsString: JSON.stringify(process.env.SKYFLOW_CREDENTIALS) - }; - ``` - -5. **Environment variables** - If no credentials are explicitly provided, the SDK automatically looks for the SKYFLOW_CREDENTIALS environment variable. Use to avoid hardcoding credentials in source code. This variable must return an object like one of the examples above. - -> [!NOTE] -> Only one type of credential can be used at a time. If multiple credentials are provided, the last one added will take precedence. - -### Generate bearer tokens for authentication & authorization - -Generate and manage bearer tokens to authenticate API calls. This section covers options for scoping to certain roles, passing context, and signing data tokens. - -#### Generate a bearer token - -Generate service account tokens using the [Service Account](https://github.com/skyflowapi/skyflow-node/tree/v2/src/service-account) Node package with a service account credentials file provided when a service account is created. Tokens generated by this module are valid for 60 minutes and can be used to make API calls to the [Data](https://docs.skyflow.com/record/) and [Management](https://docs.skyflow.com/management/) APIs, depending on the permissions assigned to the service account. - -##### `generateBearerToken(filepath)` - -The `generateBearerToken(filepath)` function takes the `credentials.json` file path for token generation. - -```ts -import { generateBearerToken, BearerTokenOptions } from 'skyflow-node'; - -const options: BearerTokenOptions = { - roleIds: ['roleId1'], // optional: scope to specific roles - ctx: 'user_12345', // optional: embed context - logLevel: LogLevel.ERROR, -}; - -generateBearerToken('path/to/credentials.json', options) - .then(response => { - const bearerToken = response.accessToken; - }) - .catch(error => { /* handle error */ }); -``` - -##### `generateBearerTokenFromCreds(credentialsString)` - -Use `generateBearerTokenFromCreds` when your credentials JSON is available as a string (e.g. from a secret manager) rather than a file on disk. It accepts the same `BearerTokenOptions` as `generateBearerToken`. - -```ts -import { generateBearerTokenFromCreds, BearerTokenOptions } from 'skyflow-node'; - -const credentialsString = JSON.stringify(require('./credentials.json')); -const options: BearerTokenOptions = { roleIds: ['roleId1'] }; - -const response = await generateBearerTokenFromCreds(credentialsString, options); -const bearerToken = response.accessToken; -``` - -**`BearerTokenOptions` type:** - -```ts -type BearerTokenOptions = { - roleIds?: string[]; // Scope the token to specific service-account roles - ctx?: string | Record; // Context value(s) embedded in the token - logLevel?: LogLevel; // Override SDK log level for this call - tokenUri?: string; // Override the token endpoint URL from the credentials file -}; -``` - - -**Checking whether a token has expired — `isExpired`:** - -```ts -import { isExpired } from 'skyflow-node'; - -// Returns true if the JWT bearer token is expired or will expire within the threshold -const expired: boolean = isExpired(bearerToken); -if (expired) { - // regenerate before making API calls - const { accessToken } = await generateBearerToken('path/to/credentials.json'); -} -``` - -> [!TIP] -> See the full example in the samples directory: [token-generation-example.ts](http://github.com/skyflowapi/skyflow-node/blob/v2/samples/service-account/token-generation-example.ts) - -#### Generate bearer tokens scoped to certain roles - -Generate bearer tokens with access limited to a specific role by specifying the appropriate `roleIds` when using a service account with multiple roles. Use this to limit access for services with multiple responsibilities, such as segregating access for billing and analytics. Generated bearer tokens are valid for 60 minutes and can only execute operations permitted by the permissions associated with the designated role. - -```ts -const options: BearerTokenOptions = { - roleIds: ['roleId1', 'roleId2'], -}; -``` - -> [!TIP] -> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/service-account/scoped-token-generation-example.ts) -> See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. - -#### Generate bearer tokens with `ctx` for context-aware authorization - -Embed context values into a bearer token during generation so you can reference those values in your policies. This enables more flexible access controls, such as tracking end-user identity when making API calls using service accounts, and facilitates using signed data tokens during detokenization. - -Generate bearer tokens containing context information using a service account with the `context_id` identifier. Context information is represented as a JWT claim in a Skyflow-generated bearer token. Tokens generated from such service accounts include a `context_identifier` claim, are valid for 60 minutes, and can be used to make API calls to the Data and Management APIs, depending on the service account's permissions. - -The `ctx` parameter accepts either a **string** or a **JSON object**: - -**String context** — use when your policy references a single context value: - -```typescript -const options = { - ctx: 'user_12345', -}; -const response = await generateBearerToken(filepath, options); -``` - -**JSON object context** — use when your policy needs multiple context values for conditional data access. Each key in the `ctx` object maps to a Skyflow CEL policy variable under `request.context.*`: - -```typescript -const options = { - ctx: { - role: 'admin', - department: 'finance', - user_id: 'user_12345', - }, -}; -const response = await generateBearerToken(filepath, options); -``` - -With the object above, your Skyflow policies can reference `request.context.role`, `request.context.department`, and `request.context.user_id` to make conditional access decisions. - -You can also set the `context` field on credentials for automatic token generation: - -```typescript -// String context on credentials -const credentials: PathCredentials = { - path: 'path/to/credentials.json', - context: 'user_12345', -}; - -// JSON object context on credentials -const credentials: PathCredentials = { - path: 'path/to/credentials.json', - context: { - role: 'admin', - department: 'finance', - }, -}; -``` - -> [!TIP] -> See the full example in the samples directory: [token-generation-with-context-example.ts](samples/service-account/token-generation-with-context-example.ts) -> See Skyflow's [context-aware authorization](https://docs.skyflow.com) and [conditional data access](https://docs.skyflow.com) docs for policy variable syntax like `request.context.*`. - -#### Generate signed data tokens - -Digitally sign data tokens with a service account's private key to add an extra layer of protection. Skyflow generates data tokens when sensitive data is inserted into the vault. Detokenize signed tokens only by providing the signed data token along with a bearer token generated from the service account's credentials. The service account must have the necessary permissions and context to successfully detokenize the signed data tokens. - -Two variants are available — choose based on how you supply credentials: - -```ts -import { - generateSignedDataTokens, - generateSignedDataTokensFromCreds, - SignedDataTokensOptions, -} from 'skyflow-node'; - -const options: SignedDataTokensOptions = { - dataTokens: ['dataToken1', 'dataToken2'], // required: tokens to sign - timeToLive: 90, // optional: seconds until expiry (default 60) - ctx: 'user_12345', // optional: context embedded in token -}; - -// From a credentials file on disk: -const response = await generateSignedDataTokens('path/to/credentials.json', options); - -// From a credentials JSON string (e.g. from a secret manager): -const credentialsString = JSON.stringify(require('./credentials.json')); -const response = await generateSignedDataTokensFromCreds(credentialsString, options); - -response.forEach(({ token, signedToken }) => { - console.log('Original token:', token); - console.log('Signed token:', signedToken); -}); -``` - -The `ctx` parameter also accepts a JSON object for multi-value context: - -```typescript -const options: SignedDataTokensOptions = { - dataTokens: ['dataToken1'], - ctx: { role: 'analyst', department: 'research' }, -}; -``` - -**`SignedDataTokensOptions` type:** - -```ts -type SignedDataTokensOptions = { - dataTokens: string[]; // Data tokens to sign (required) - timeToLive?: number; // Seconds until token expiry (default 60) - ctx?: string | Record; // Context value(s) embedded in the token - logLevel?: LogLevel; // Override SDK log level for this call - tokenUri?: string; // Override the token endpoint URL from credentials -}; -``` - -> [!TIP] -> See the full example in the samples directory: [signed-token-generation-example.ts](samples/service-account/signed-token-generation-example.ts) -> See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. - -## Client management - -After initializing the `Skyflow` client, you can add, update, retrieve, and remove vault and connection configurations at any time without recreating the client. For the full list of methods, see [Client management methods](docs/api_reference.md#client-management-methods) in the API reference. - -## Using the client in production - -**Initialize once and reuse.** `new Skyflow(config)` returns a long-lived client that caches tokens per vault. Construct it once at startup (e.g. as a module-level singleton) and reuse it across requests. Recreating the client on every call discards these caches and forces unnecessary token regeneration. - -```ts -// At application startup -const skyflowClient = new Skyflow(skyflowConfig); - -// Reuse `skyflowClient` for the lifetime of the process -``` - -**Bearer token refresh is automatic.** When you authenticate with a service-account credentials file/string (or API key), the SDK caches the generated bearer token and regenerates it automatically once it expires. You don't need to manage token lifecycle yourself. (For the rare expire-mid-request case, see [Bearer token expiration edge cases](#bearer-token-expiration-edge-cases).) - -**Runtime configuration changes are not concurrency-safe.** Methods that mutate client state at runtime — `addVaultConfig`, `updateVaultConfig`, `removeVaultConfig`, the `*ConnectionConfig` methods, and `updateSkyflowCredentials` — should be called during setup, not concurrently with in-flight requests. Once configured, the client is safe to reuse across concurrent calls. - -**Timeouts and retries.** The SDK does not currently expose request timeout or automatic-retry configuration. Wrap your SDK calls with your own timeout/retry logic at the application layer if needed. - -## Logging - -The SDK provides useful logging. By default, the logging level is set to `LogLevel.ERROR`. Change this by setting the `logLevel` in Skyflow Config while creating the Skyflow Client as shown below: - -Currently, the following five log levels are supported: - -- `DEBUG`: - When `LogLevel.DEBUG` is passed, logs at all levels will be printed (DEBUG, INFO, WARN, ERROR). -- `INFO`: - When `LogLevel.INFO` is passed, INFO logs for every event that occurs during SDK flow execution will be printed, along with WARN and ERROR logs. -- `WARN`: - When `LogLevel.WARN` is passed, only WARN and ERROR logs will be printed. -- `ERROR`: - When `LogLevel.ERROR` is passed, only ERROR logs will be printed. -- `OFF`: - `LogLevel.OFF` can be used to turn off all logging from the Skyflow Node SDK. - -> [!NOTE] -> The ranking of logging levels is as follows: `DEBUG` < `INFO` < `WARN` < `ERROR` < `OFF`. - -### Example `skyflowConfig.logLevel: LogLevel.INFO` - -```ts -const skyflowConfig: SkyflowConfig = { - vaultConfigs: [vaultConfig], // Add the Vault configuration - skyflowCredentials: skyflowCredentials, // Use Skyflow credentials if no token is passed - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production environment. -}; - -const skyflowClient: Skyflow = new Skyflow(skyflowConfig); -``` - -## Error handling - -### Catching `SkyflowError` instances - -Wrap your calls to the Skyflow SDK in try/catch blocks as a best practice. Use the `SkyflowError` type to identify errors coming from Skyflow versus general request/response errors. - -```ts -try { - // ...call the Skyflow SDK -} catch (error) { - // catch an error, identify if it is a SkyflowError - if (error instanceof SkyflowError) { - console.error("Skyflow Specific Error:", { - code: error.error?.httpCode, - message: error.message, - requestId: error.error?.requestId, - details: error.error?.details, - }); - } else { - console.error("Unexpected Error:", JSON.stringify(error)); - } -} -``` - -### Per-record errors (`SkyflowRecordError`) - -Some operations — such as `insert` with `continueOnError: true` and `detokenize` — can partially succeed. In these cases the response object contains an `errors` array alongside the successful results. Each entry in `errors` is a `SkyflowRecordError`: - -```ts -import { InsertResponse, SkyflowRecordError } from 'skyflow-node'; - -const response: InsertResponse = await skyflowClient.vault(vaultId).insert(request, options); - -if (response.errors && response.errors.length > 0) { - response.errors.forEach((err: SkyflowRecordError) => { - console.error({ - requestId: err.requestId, // x-request-id from the server response - httpCode: err.httpCode, // HTTP status code for this record - requestIndex: err.requestIndex, // Index of the failed record in the input array - error: err.error, // Error description string - token: err.token, // Token (detokenize errors only) - }); - }); -} -``` - -`SkyflowRecordError` shape: - -```ts -interface SkyflowRecordError { - error: string; - requestId: string | null; - httpCode?: string | number | null; - requestIndex?: number | null; - token?: string | null; // present on detokenize errors -} -``` - -### Bearer token expiration edge cases - -When using bearer tokens for authentication and API requests, a token may expire after verification but before the actual API call completes. This causes the request to fail unexpectedly. An error from this edge case looks like this: - -```txt -message: Authentication failed. Bearer token is expired. Use a valid bearer token. See https://docs.skyflow.com/api-authentication/ -``` - -If you encounter this kind of error, retry the request. During the retry the SDK detects that the previous bearer token has expired and generates a new one for the current and subsequent requests. - -> [!TIP] -> See the full example in the samples directory: [bearer-token-expiry-example.ts](samples/service-account/bearer-token-expiry-example.ts) -> See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. - -## Troubleshooting - -Most first-run problems come from configuration mismatches. Every error thrown by the SDK is a `SkyflowError` exposing `httpCode`, `message`, and `details` — inspect these first (see [Error handling](#error-handling)). - -| Symptom | Likely cause | Fix | -|---|---|---| -| `npm install skyflow-node` fails | Unsupported Node version | Requires Node v12.22.12 or above. | -| Connection/DNS failures, or 404 on every call | Wrong `clusterId` | `clusterId` is the first segment of your vault URL: `https://{clusterId}.vault.skyflowapis.com`. | -| Requests hit the wrong host / unexpected auth failures | Wrong `env` | Match `env` to where your vault runs (`Env.PROD`, `Env.SANDBOX`, `Env.DEV`, `Env.STAGE`). | -| `401 Unauthorized` | Invalid or expired credentials | Verify your API key or service-account credentials. Regenerate if needed. | -| `403 Forbidden` | Service account lacks permission | Grant the service account a role with the required permissions, or use a [scoped token](#generate-bearer-tokens-scoped-to-certain-roles). | -| `404` referencing a table or column | Name mismatch | Confirm table and column names match your vault schema exactly (case-sensitive). | -| Vault not found / 404 with a valid `clusterId` | Wrong `vaultId` | Copy `vaultId` from the vault's details page in Skyflow Studio. | -| `Authentication failed. Bearer token is expired.` | Token expired mid-request | Retry the request; the SDK regenerates the token. See [Bearer token expiration edge cases](#bearer-token-expiration-edge-cases). | -| Unexpected credential is used | Multiple credential types provided | Only one credential type is used at a time; the last one added takes precedence. | +## Which package do I want? -If you're stuck, set `logLevel: LogLevel.DEBUG` during development for detailed SDK logs (see [Logging](#logging)). +| Package | npm | README | Vault Type | Version line | +|---|---|---|---|---| +| **skyvault** | [`skyflow-node`](https://www.npmjs.com/package/skyflow-node) | [skyvault/README.md](skyvault/README.md) | Privacy DB | 2.x | +| **flowvault** | [`skyflow-flowvault-node`](https://www.npmjs.com/package/skyflow-flowvault-node) | [flowvault/README.md](flowvault/README.md) | Flow DB | 1.x | -## TypeScript types reference +`flowvault` shares auth/client setup with `skyvault` — both depend on the `common` workspace. Install exactly one of the two packages. -For the full reference of all exported types, interfaces, enums, request/response fields, and options setters, see the **[API Reference](docs/api_reference.md)**. +> **The two packages are versioned independently.** `flowvault` is a new SDK starting at `1.0.0`; its lower version number reflects a first release, not an older or lesser SDK than `skyvault` 2.x. Upgrade each on its own version line. -The most commonly imported TypeScript types are: +> Migrating from v1? See skyvault's **[Migration Guide](docs/migrate_to_v2.md)**. V1 is in maintenance mode and will reach End of Life on October 31, 2026. -```ts -import { - // Credential sub-types (for stricter typing) - ApiKeyCredentials, - TokenCredentials, - PathCredentials, // supports optional tokenUri - StringCredentials, // supports optional tokenUri +## Repository layout - // Request item shapes - DetokenizeData, // { token: string; redactionType?: RedactionType } - TokenizeRequestType, // { value: string; columnGroup: string } +The root `package.json` (`private: true`) aggregates these npm workspaces: - // Detect configuration and rule shapes - DetectConfiguration, // inline Detect configuration: { detect?, fileMapping?, media? } - RedactionLevel, // { entityName?, redactionType? } (skyvault); the flowvault SDK has { tokenGroupName?, redactionPattern? } - PollOptions, // { waitTime?: number; maxAttempts?: number } +- `common/` — shared client, credentials, config, Detect, and error-handling code used by both `skyvault` and `flowvault` +- `skyvault/` — the `skyflow-node` SDK ([README](skyvault/README.md)) +- `flowvault/` — the `skyflow-flowvault-node` SDK ([README](flowvault/README.md)) +- `samples/` — runnable examples for `skyvault` ([README](samples/README.md)); `flowvault` samples live in [`flowvault/samples/`](flowvault/samples/README.md) - // Response record shapes - InsertResponseType, // { skyflowId: string; [field: string]: unknown } - GetResponseData, // { [field: string]: unknown } - QueryResponseType, // { [field: string]: unknown } - SkyflowRecordError, // shape of entries in response.errors[] -} from 'skyflow-node'; -``` +## Documentation -## Security -### Reporting a Vulnerability +- [skyvault API Reference](docs/api_reference.md) — full list of request classes, options, response classes, enums, and service-account functions +- [Detect API Reference](docs/api_reference.md#detect-api) — request, response, configuration and enum types for Detect, shared by both SDKs +- [Migrate from v1 to v2](docs/migrate_to_v2.md) +- [Advanced initialization](docs/advanced_initialization.md) +- [Authentication credentials](docs/auth_credentials.md) -If you discover a potential security issue in this project, reach out to us at [security@skyflow.com](mailto:security@skyflow.com). +## Reporting a Vulnerability -Don't create public GitHub issues or Pull Requests, as malicious actors could potentially view them. +If you discover a potential security issue in this project, please reach out to us at **security@skyflow.com**. Please do not create public GitHub issues or Pull Requests, as malicious actors could potentially view them. diff --git a/common/src/detect/internal/request-mapper.ts b/common/src/detect/internal/request-mapper.ts index 502bad93..e0911894 100644 --- a/common/src/detect/internal/request-mapper.ts +++ b/common/src/detect/internal/request-mapper.ts @@ -10,7 +10,9 @@ import { Detect as DetectV2, DetectConfigV2, DetectCustomTypes, + DetectCustomTypesIncludePrefix, DetectEntity, + DetectEntityIncludePrefix, DetectGuardrailsRequestV2, DetectObjectEntity, DocumentV2, @@ -39,6 +41,7 @@ import { FileDataFormat, FileMapping, Image, + IncludePrefix, Media, ObjectEntity, RedactionRule, @@ -242,12 +245,24 @@ function toDetect(detect: Detect): DetectV2 { }); } +/** + * The REST enum has only YES and NO. INCLUDE_PREFIX_UNSPECIFIED (or an unset field) is omitted from the + * request so the server applies its default, which is YES. + */ +function toIncludePrefix(includePrefix?: IncludePrefix): DetectEntityIncludePrefix | undefined { + return includePrefix === undefined || includePrefix === IncludePrefix.INCLUDE_PREFIX_UNSPECIFIED + ? undefined + : (includePrefix as string as DetectEntityIncludePrefix); +} + function toEntity(entity: Entity): DetectEntity { return compact({ entityType: entity.entityType, deidentificationType: entity.deidentificationType, destination: entity.destination, transformation: entity.transformation ? toTransformation(entity.transformation) : undefined, + includePrefix: toIncludePrefix(entity.includePrefix), + prefixOverride: entity.prefixOverride, }); } @@ -259,7 +274,14 @@ function toTransformation(transformation: Transformation): EntityTransformation } function toCustomType(customType: CustomType): DetectCustomTypes { - return compact({ label: customType.label, match: customType.match }); + return compact({ + label: customType.label, + match: customType.match, + deidentificationType: customType.deidentificationType, + destination: customType.destination, + includePrefix: toIncludePrefix(customType.includePrefix) as DetectCustomTypesIncludePrefix | undefined, + prefixOverride: customType.prefixOverride, + }); } function toObjectEntity(objectEntity: ObjectEntity): DetectObjectEntity { diff --git a/common/src/detect/types.ts b/common/src/detect/types.ts index fe24c4f5..fbc94925 100644 --- a/common/src/detect/types.ts +++ b/common/src/detect/types.ts @@ -92,6 +92,17 @@ export enum DeidentificationType { MASK = "MASK", } +/** + * Whether the entity label is included as a prefix in a generated vault token. Applies to + * `VAULT_TOKEN` only. `INCLUDE_PREFIX_UNSPECIFIED` leaves the field unset so the server applies + * its default, which is `YES`. `NO` returns the raw vault token, which cannot be re-identified. + */ +export enum IncludePrefix { + INCLUDE_PREFIX_UNSPECIFIED = "INCLUDE_PREFIX_UNSPECIFIED", + YES = "YES", + NO = "NO", +} + /** * Contract for the source of a `deidentifyFile` input and the delivery of a run's outputs. * Each SDK exposes its own `DeidentifySourceType` enum with the values its vault supports @@ -226,12 +237,37 @@ export interface Entity { */ destination?: string; transformation?: Transformation; + /** + * Used only with `VAULT_TOKEN`: whether the entity label prefixes the token, as in + * `[NAME_]`. Server default `YES`. `NO` returns the raw vault token, which cannot + * be re-identified. `INCLUDE_PREFIX_UNSPECIFIED` leaves the field unset. + */ + includePrefix?: IncludePrefix; + /** + * Used only with `VAULT_TOKEN`: replaces the entity-type prefix in the token, for example + * `NAAM` gives `[NAAM_]`. Not valid together with `includePrefix: NO`. + */ + prefixOverride?: string; } -/** A regex-defined entity type. `label` is reported as the entity type of its matches. */ +/** + * A regex-defined entity type. `label` is reported as the entity type of its matches and is + * the default token prefix. Matches are de-identified like any other entity. + */ export interface CustomType { label?: string; match?: string[]; + /** Defaults to `ENTITY_UNIQUE_COUNTER` on the server. */ + deidentificationType?: DeidentificationType; + /** + * Required with `VAULT_TOKEN`: where matches are stored. A `table.column` on skyvault + * vaults, a token group name on flowvault vaults. Must exist. + */ + destination?: string; + /** Used only with `VAULT_TOKEN`: whether the label prefixes the token. Server default `YES`. */ + includePrefix?: IncludePrefix; + /** Used only with `VAULT_TOKEN`: replaces the label as the token prefix. Not valid with `includePrefix: NO`. */ + prefixOverride?: string; } /** A visual object type to detect in images and how to handle it. */ diff --git a/common/test/detect/request-mapper.test.js b/common/test/detect/request-mapper.test.js index 31bfa7db..a2ddcf2a 100644 --- a/common/test/detect/request-mapper.test.js +++ b/common/test/detect/request-mapper.test.js @@ -22,6 +22,7 @@ import { DeidentificationType, EntityType, FileDataFormat, + IncludePrefix, MaskingMethod, ObjectDeidentificationType, ObjectEntityType, @@ -86,13 +87,29 @@ describe('toDetectConfig', () => { deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, transformation: { shiftDates: { minDays: 10, maxDays: 30 } }, }, - { entityType: EntityType.NAME, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: 'pii.name' }, + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: 'pii.name', + includePrefix: IncludePrefix.YES, + prefixOverride: 'NAAM', + }, null, ], objectEntities: [{ entityType: ObjectEntityType.FACE, deidentificationType: ObjectDeidentificationType.REDACT }], restrict: ['[0-9]{3}'], skip: ['Skyflow'], - customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], + customTypes: [ + { label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }, + { + label: 'PRODUCT', + match: ['PRD-[0-9]{4}'], + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: 'productTokenGroup', + includePrefix: IncludePrefix.NO, + prefixOverride: 'PRODUKT', + }, + ], returnEntities: ReturnEntitiesType.ALL, }, fileMapping: [{ source: 'a', destination: 'b', outputFileNameSuffix: '_x', entitiesDestination: 'e', objectEntitiesDestination: 'o' }], @@ -111,12 +128,28 @@ describe('toDetectConfig', () => { detect: { entities: [ { entityType: 'DOB', deidentificationType: 'ENTITY_UNIQUE_COUNTER', transformation: { shiftDates: { minDays: 10, maxDays: 30 } } }, - { entityType: 'NAME', deidentificationType: 'VAULT_TOKEN', destination: 'pii.name' }, + { + entityType: 'NAME', + deidentificationType: 'VAULT_TOKEN', + destination: 'pii.name', + includePrefix: 'YES', + prefixOverride: 'NAAM', + }, ], objectEntities: [{ entityType: 'FACE', deidentificationType: 'REDACT' }], restrict: ['[0-9]{3}'], skip: ['Skyflow'], - customTypes: [{ label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }], + customTypes: [ + { label: 'EMPLOYEE_ID', match: ['EMP-[0-9]{6}'] }, + { + label: 'PRODUCT', + match: ['PRD-[0-9]{4}'], + deidentificationType: 'VAULT_TOKEN', + destination: 'productTokenGroup', + includePrefix: 'NO', + prefixOverride: 'PRODUKT', + }, + ], returnEntities: 'ALL', }, fileMapping: [{ source: 'a', destination: 'b', outputFileNameSuffix: '_x', entitiesDestination: 'e', objectEntitiesDestination: 'o' }], @@ -131,6 +164,34 @@ describe('toDetectConfig', () => { const out = toDetectConfig({ detect: { entities: [{ transformation: {} }] }, media: {} }, VAULT); expect(out).toEqual({ vaultId: VAULT, detect: { entities: [{ transformation: {} }] }, media: {} }); }); + test('INCLUDE_PREFIX_UNSPECIFIED and an unset includePrefix are omitted; YES and NO are sent', () => { + const entities = (includePrefix) => + toDetectConfig({ detect: { entities: [includePrefix === null ? {} : { includePrefix }] } }, VAULT).detect.entities; + expect(entities(IncludePrefix.INCLUDE_PREFIX_UNSPECIFIED)).toEqual([{}]); + expect(entities(null)).toEqual([{}]); + expect(entities(IncludePrefix.YES)).toEqual([{ includePrefix: 'YES' }]); + expect(entities(IncludePrefix.NO)).toEqual([{ includePrefix: 'NO' }]); + + const customTypes = (includePrefix) => + toDetectConfig({ detect: { customTypes: [{ label: 'ID', includePrefix }] } }, VAULT).detect.customTypes; + expect(customTypes(IncludePrefix.INCLUDE_PREFIX_UNSPECIFIED)).toEqual([{ label: 'ID' }]); + expect(customTypes(IncludePrefix.NO)).toEqual([{ label: 'ID', includePrefix: 'NO' }]); + }); + test('a custom type with only label and match carries no vault-token fields', () => { + const out = toDetectConfig({ detect: { customTypes: [{ label: 'ID', match: ['[0-9]+'] }] } }, VAULT); + expect(out.detect.customTypes).toEqual([{ label: 'ID', match: ['[0-9]+'] }]); + expect(Object.keys(out.detect.customTypes[0])).toEqual(['label', 'match']); + }); + test('every DeidentificationType maps for entities and custom types', () => { + Object.values(DeidentificationType).forEach((type) => { + const out = toDetectConfig( + { detect: { entities: [{ deidentificationType: type }], customTypes: [{ deidentificationType: type }] } }, + VAULT, + ); + expect(out.detect.entities).toEqual([{ deidentificationType: type }]); + expect(out.detect.customTypes).toEqual([{ deidentificationType: type }]); + }); + }); }); describe('toDeidentifyFileRequest', () => { diff --git a/docs/api_reference.md b/docs/api_reference.md index 7a4d2d12..ecb136bf 100644 --- a/docs/api_reference.md +++ b/docs/api_reference.md @@ -1,6 +1,6 @@ # API Reference -A reference for the public Skyflow Node SDK surface: client-management methods, request classes, options classes, response classes, enums, the Detect API (shared with the flowvault SDK), and service-account functions. For task-oriented usage and examples, see the [README](../README.md). +A reference for the public Skyflow Node SDK surface: client-management methods, request classes, options classes, response classes, enums, the Detect API (shared with the flowvault SDK), and service-account functions. For task-oriented usage and examples, see the [skyvault README](../skyvault/README.md). All properties, setters, and enum values below are taken directly from the SDK source. @@ -335,7 +335,7 @@ Deployment environment. ### `LogLevel` -`DEBUG` < `INFO` < `WARN` < `ERROR` < `OFF`. Default is `ERROR`. See [Logging](../README.md#logging). +`DEBUG` < `INFO` < `WARN` < `ERROR` < `OFF`. Default is `ERROR`. See [Logging](../skyvault/README.md#logging). ### `RedactionType` @@ -380,7 +380,7 @@ Values: `GET`, `POST`, `PUT`, `PATCH`. ## Detect API -Shared by the `skyflow-node` and `skyflow-flowvault-node` SDKs: every class, interface and enum below is importable from either package. `skyflowClient.detect()` and `skyflowClient.detect(vaultId)` return a `DetectController`. Usage and examples: [skyvault README](../README.md#detect), [flowvault README](../flowvault/README.md#detect). +Shared by the `skyflow-node` and `skyflow-flowvault-node` SDKs: every class, interface and enum below is importable from either package. `skyflowClient.detect()` and `skyflowClient.detect(vaultId)` return a `DetectController`. Usage and examples: [skyvault README](../skyvault/README.md#detect), [flowvault README](../flowvault/README.md#detect). | Method | Request | Response | |--------|---------|----------| @@ -548,6 +548,8 @@ Inline Detect policy for `deidentifyString` and `deidentifyFile`. A plain interf | `deidentificationType` | `DeidentificationType` | How the value is replaced. Server default `ENTITY_UNIQUE_COUNTER`. | | `destination` | `string` | For `VAULT_TOKEN`: where the value is stored. A `table.column` on skyvault vaults, a token group name on flowvault vaults. Must already exist. | | `transformation` | [`Transformation`](#transformation-and-shiftdates) | Date shifting. | +| `includePrefix` | `IncludePrefix` | For `VAULT_TOKEN`: whether the entity label prefixes the token (`[NAME_]`). Server default `YES`. `NO` returns the raw vault token, which cannot be re-identified. `INCLUDE_PREFIX_UNSPECIFIED` leaves the field unset. | +| `prefixOverride` | `string` | For `VAULT_TOKEN`: replaces the entity-type prefix, for example `NAAM` gives `[NAAM_]`. Not valid with `includePrefix: NO`. | ### `Transformation` and `ShiftDates` @@ -557,8 +559,12 @@ Inline Detect policy for `deidentifyString` and `deidentifyFile`. A plain interf | Field | Type | Description | |---|---|---| -| `label` | `string` | Becomes the entity type on the response. | +| `label` | `string` | Becomes the entity type on the response and the default token prefix. | | `match` | `string[]` | Regexes that define the type. | +| `deidentificationType` | `DeidentificationType` | How matches are replaced. Server default `ENTITY_UNIQUE_COUNTER`. | +| `destination` | `string` | For `VAULT_TOKEN`: where matches are stored. A `table.column` on skyvault vaults, a token group name on flowvault vaults. Required with `VAULT_TOKEN`. | +| `includePrefix` | `IncludePrefix` | For `VAULT_TOKEN`: whether the label prefixes the token. Server default `YES`. `NO` returns the raw vault token, which cannot be re-identified. | +| `prefixOverride` | `string` | For `VAULT_TOKEN`: replaces the label as the token prefix. Not valid with `includePrefix: NO`. | ### `ObjectEntity` @@ -664,6 +670,7 @@ All string enums whose values match the wire values. |------|--------| | `EntityType` | the 70 entity types (`NAME`, `EMAIL_ADDRESS`, `SSN`, `CREDIT_CARD`, `DOB`, ...), plus `ALL` and `NONE` | | `DeidentificationType` | `ENTITY_UNIQUE_COUNTER` _(server default)_, `ENTITY_ONLY`, `VAULT_TOKEN`, `MASK`, `UNKNOWN` | +| `IncludePrefix` | `YES` _(server default)_, `NO`, `INCLUDE_PREFIX_UNSPECIFIED` _(field omitted)_. `VAULT_TOKEN` only. | | `DeidentifySourceType` | Per SDK: skyvault `BASE64`, `SKYFLOW_ID`, `PRESIGNED_URL`; flowvault `BASE64`, `PRESIGNED_URL`. Input source for `deidentifyFile`, and the `outputType` of `DeidentifyFileResponse` and `GetRunResponse`. | | `ReidentifySourceType` | Per SDK, `BASE64`, `PRESIGNED_URL` on both. Input source for `reidentifyFile` and the `outputType` of `ReidentifyFileResponse`. | | `FileDataFormat` | `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML`, `ZIP` | @@ -680,7 +687,7 @@ All string enums whose values match the wire values. ## Service account functions -All importable from `'skyflow-node'`. See [Authentication & authorization](../README.md#authentication--authorization) for full usage of `generateBearerToken`, `generateBearerTokenFromCreds`, and `generateSignedDataTokens`. +All importable from `'skyflow-node'`. See [Authentication & authorization](../skyvault/README.md#authentication--authorization) for full usage of `generateBearerToken`, `generateBearerTokenFromCreds`, and `generateSignedDataTokens`. ### `isExpired(token)` diff --git a/docs/migrate_to_v2.md b/docs/migrate_to_v2.md index 60a0f076..7bd74101 100644 --- a/docs/migrate_to_v2.md +++ b/docs/migrate_to_v2.md @@ -131,7 +131,7 @@ In v2, with the introduction of TypeScript support, you can now pass an **Insert The response will be of type InsertResponse, which contains insertedFields and errors. -**Note:** Similar patterns apply to other operations like Get, Update, Delete. See the [README](../README.md) for complete examples. +**Note:** Similar patterns apply to other operations like Get, Update, Delete. See the [skyvault README](../skyvault/README.md) for complete examples. #### V1 (Old) - Request Building diff --git a/flowvault/README.md b/flowvault/README.md index 62c0817e..a1653467 100644 --- a/flowvault/README.md +++ b/flowvault/README.md @@ -5,7 +5,7 @@ for the v2 vault data plane. With it you can securely insert, retrieve, update, detokenize, tokenize, and attach files to records in a Flow DB vault, and de-identify and re-identify sensitive data in text and files with Skyflow Detect. -> This is a **separate product** from `skyflow-node` (the SkyVault SDK). Install exactly one: +> This is a **separate product** from `skyflow-node` (the [SkyVault SDK](../skyvault/README.md), for Privacy DB vaults). Install exactly one: > `npm i skyflow-flowvault-node` **or** `npm i skyflow-node`. A lower `flowvault` version number > does not mean it is older — the two products version independently. @@ -348,9 +348,9 @@ console.log(inlineResponse.metrics?.wordCount); | Type | Fields | Notes | |------|--------|-------| | `Detect` | `entities`, `objectEntities`, `restrict`, `skip`, `customTypes`, `returnEntities` | `restrict` and `skip` are regexes that are always, respectively never, treated as sensitive. `returnEntities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. | -| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` is used only with `VAULT_TOKEN`. | +| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation`, `includePrefix`, `prefixOverride` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination`, `includePrefix` and `prefixOverride` apply only with `VAULT_TOKEN`: `includePrefix` is `YES` (server default), `NO` (raw token, cannot be re-identified) or `INCLUDE_PREFIX_UNSPECIFIED` (field omitted); `prefixOverride` replaces the entity-type prefix in the token and is not valid with `includePrefix` `NO`. | | `Transformation` / `ShiftDates` | `shiftDates`; `minDays`, `maxDays` | Shifts detected dates by a random number of days within the range. | -| `CustomType` | `label`, `match` | A regex-defined entity type. `label` is reported as the entity type of its matches. | +| `CustomType` | `label`, `match`, `deidentificationType`, `destination`, `includePrefix`, `prefixOverride` | A regex-defined entity type. `label` is reported as the entity type of its matches and is the default token prefix. The remaining fields work as on `Entity`; `destination` is required with `VAULT_TOKEN`. | | `ObjectEntity` | `entityType`, `deidentificationType` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. | | `FileMapping` | `source`, `destination`, `outputFileNameSuffix`, `entitiesDestination`, `objectEntitiesDestination` | Where the processed file and the entity files are written. File inputs only. | | `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. | diff --git a/flowvault/api-report/skyflow-flowvault-node.api.md b/flowvault/api-report/skyflow-flowvault-node.api.md index 0fbee5f2..0685496f 100644 --- a/flowvault/api-report/skyflow-flowvault-node.api.md +++ b/flowvault/api-report/skyflow-flowvault-node.api.md @@ -28,6 +28,7 @@ import { BaseUploadFilesResponse } from 'skyflow-common'; import { BaseVaultEntry } from 'skyflow-common'; import { Credentials } from 'skyflow-common'; import { LogLevel } from 'skyflow-common'; +import { RedactionRule } from 'skyflow-common'; import { SkyflowConfig } from 'skyflow-common'; import { SkyflowError } from 'skyflow-common'; import { SkyflowRecordError } from 'skyflow-common'; @@ -51,6 +52,14 @@ export enum CustomHeaderKey { SKYFLOW_ACCOUNT_NAME = "x-skyflow-account-name" } +// @public +export enum DeidentifySourceType { + // (undocumented) + BASE64 = "BASE64", + // (undocumented) + PRESIGNED_URL = "PRESIGNED_URL" +} + // @public export class DeleteFilesOptions extends FlowVaultOptions { constructor(); @@ -429,6 +438,24 @@ export interface QueryResponseMetadata { columns: string[]; } +// @public +export interface RedactionLevel extends RedactionRule { + // (undocumented) + entityName?: never; + redactionPattern?: string; + // (undocumented) + redactionType?: never; + tokenGroupName?: string; +} + +// @public +export enum ReidentifySourceType { + // (undocumented) + BASE64 = "BASE64", + // (undocumented) + PRESIGNED_URL = "PRESIGNED_URL" +} + // @public export class RequestContext { constructor(operation: string, batchIndex?: number, totalBatches?: number); diff --git a/skyvault/README.md b/skyvault/README.md index 55cb491f..5bc7ba73 100644 --- a/skyvault/README.md +++ b/skyvault/README.md @@ -2,7 +2,9 @@ > **This is the current, recommended version of the Skyflow SDK.** V2.1.0 brings flexible auth, multi-vault support, builder patterns, native data types, and rich error diagnostics. > -> Migrating from v1? See the **[Migration Guide](docs/migrate_to_v2.md)** for step-by-step instructions. V1 is in maintenance mode and will reach End of Life on October 31, 2026. +> Meant for **Privacy DB** vaults. Working with a Flow DB vault? Use the [FlowVault SDK](../flowvault/README.md) (`skyflow-flowvault-node`) instead. +> +> Migrating from v1? See the **[Migration Guide](../docs/migrate_to_v2.md)** for step-by-step instructions. V1 is in maintenance mode and will reach End of Life on October 31, 2026. Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow SDK for Node.js, Deno, Bun, and Cloudflare Workers. @@ -85,12 +87,12 @@ Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow The Skyflow SDK enables you to connect to your Skyflow Vault(s) to securely handle sensitive data at rest, in-transit, and in-use. > [!TIP] -> Looking for the full list of request parameters, options setters, response fields, enums, and Detect helper classes? See the **[API Reference](docs/api_reference.md)**. +> Looking for the full list of request parameters, options setters, response fields, enums, and Detect helper classes? See the **[API Reference](../docs/api_reference.md)**. > [!IMPORTANT] > This readme documents SDK version 2. > For version 1 see the [v1.14.2 README](https://github.com/skyflowapi/skyflow-node/tree/1.14.2). -> For more information on how to migrate see [MIGRATE_TO_V2.md](docs/migrate_to_v2.md). +> For more information on how to migrate see [MIGRATE_TO_V2.md](../docs/migrate_to_v2.md). ## Installation @@ -197,13 +199,13 @@ const skyflowConfig: SkyflowConfig = { const skyflowClient: Skyflow = new Skyflow(skyflowConfig); ``` -See [docs/advanced_initialization.md](docs/advanced_initialization.md) for advanced initialization examples including multiple vaults and different credential types. +See [docs/advanced_initialization.md](../docs/advanced_initialization.md) for advanced initialization examples including multiple vaults and different credential types. ### Insert data into the vault, get tokens back Insert data into your vault using the `insert` method. Set `insertOptions.setReturnTokens(true)` to ensure values are tokenized in the response. -Create an insert request with the [`InsertRequest`](docs/api_reference.md#insertrequest) class, which includes the values to be inserted as a list of records. +Create an insert request with the [`InsertRequest`](../docs/api_reference.md#insertrequest) class, which includes the values to be inserted as a list of records. Below is a simple example to get started. See the [Insert and tokenize data](#insert-and-tokenize-data-insertrequest) section for advanced options. @@ -226,7 +228,7 @@ const insertResponse = await skyflowClient console.log("Insert response:", insertResponse); ``` -Returns an [`InsertResponse`](docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setReturnTokens(true)`, each entry includes the `skyflowId` and a token per column: +Returns an [`InsertResponse`](../docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setReturnTokens(true)`, each entry includes the `skyflowId` and a token per column: ```text Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '5391-4629-3722-7102', cardholder_name: '0f6b8a2c-90ab-4cde-9def-567890abcdef' }], errors: null } @@ -234,7 +236,7 @@ Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f ## Upgrade from v1 to v2 -Upgrade from `skyflow-node` v1 using the dedicated guide in [docs/migrate_to_v2.md](docs/migrate_to_v2.md). +Upgrade from `skyflow-node` v1 using the dedicated guide in [docs/migrate_to_v2.md](../docs/migrate_to_v2.md). ## Vault @@ -265,7 +267,7 @@ const response: InsertResponse = await skyflowClient console.log('Insert response:', response); ``` -Returns an [`InsertResponse`](docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setContinueOnError(true)`, `insertedFields` contains successful records and `errors` contains per-record failures: +Returns an [`InsertResponse`](../docs/api_reference.md#insertresponse) (`insertedFields`, `errors`). With `setContinueOnError(true)`, `insertedFields` contains successful records and `errors` contains per-record failures: ```text Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', '': '', '': '' }], errors: null } @@ -278,7 +280,7 @@ Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f Set the `continueOnError` flag to `true` to allow insert operations to proceed despite encountering partial errors. > [!TIP] -> See the full example in the samples directory: [insert-continue-on-error.ts](samples/vault-api/insert-continue-on-error.ts) +> See the full example in the samples directory: [insert-continue-on-error.ts](../samples/vault-api/insert-continue-on-error.ts) #### Upsert request @@ -290,13 +292,13 @@ Turn an insert into an 'update-or-insert' operation using the upsert option. The insertOptions.setUpsertColumn("cardholder_name"); ``` -For all `InsertOptions` setters and `TokenMode` values, see [InsertOptions](docs/api_reference.md#insertoptions) in the API reference. +For all `InsertOptions` setters and `TokenMode` values, see [InsertOptions](../docs/api_reference.md#insertoptions) in the API reference. ### Detokenize: `.detokenize(request, options)` Convert tokens back into plaintext values (or masked values) using the `.detokenize()` method. Detokenization accepts tokens and returns values. -Create a detokenization request with the [`DetokenizeRequest`](docs/api_reference.md#detokenizerequest) class, which requires a list of tokens and column groups as input. +Create a detokenization request with the [`DetokenizeRequest`](../docs/api_reference.md#detokenizerequest) class, which requires a list of tokens and column groups as input. Provide optional parameters such as the redaction type and the option to continue on error. @@ -325,18 +327,18 @@ const response: DetokenizeResponse = await skyflowClient console.log("Detokenization response:", response); ``` -Returns a [`DetokenizeResponse`](docs/api_reference.md#detokenizeresponse) (`detokenizedFields`, `errors`): +Returns a [`DetokenizeResponse`](../docs/api_reference.md#detokenizeresponse) (`detokenizedFields`, `errors`): ```text Detokenization response: { detokenizedFields: [{ token: '5391-4629-3722-7102', value: '4111111111111112', type: 'STRING' }], errors: null } ``` > [!TIP] -> See the full example in the samples directory: [detokenzie-records.ts](samples/vault-api/detokenzie-records.ts) +> See the full example in the samples directory: [detokenzie-records.ts](../samples/vault-api/detokenzie-records.ts) ### Get Record(s): `.get(request, options)` -Retrieve data using Skyflow IDs or unique column values with the get method. Create a get request with the [`GetRequest`](docs/api_reference.md#getrequest) class, specifying parameters such as the table name, redaction type, Skyflow IDs, column names, and column values. +Retrieve data using Skyflow IDs or unique column values with the get method. Create a get request with the [`GetRequest`](../docs/api_reference.md#getrequest) class, specifying parameters such as the table name, redaction type, Skyflow IDs, column names, and column values. > [!NOTE] > You can't use both Skyflow IDs and column name/value pairs in the same request. Use the `GetOptions` class to specify whether to return tokens. @@ -358,7 +360,7 @@ const response: GetResponse = await skyflowClient console.log("Get response:", response); ``` -Returns a [`GetResponse`](docs/api_reference.md#getresponse) (`data`, `errors`): +Returns a [`GetResponse`](../docs/api_reference.md#getresponse) (`data`, `errors`): ```text Get response: { data: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '4111111111111112', cardholder_name: 'John Doe' }], errors: null } @@ -414,7 +416,7 @@ getOptions.setReturnTokens(true); // Set to `true` to get tokens ``` > [!TIP] -> See the full example in the samples directory: [get-records.ts](samples/vault-api/get-records.ts) +> See the full example in the samples directory: [get-records.ts](../samples/vault-api/get-records.ts) #### Get by column name and column values @@ -429,7 +431,7 @@ const getRequest: GetColumnRequest = new GetColumnRequest( ``` > [!TIP] -> See the full example in the samples directory: [get-column-values.ts](samples/vault-api/get-column-values.ts) +> See the full example in the samples directory: [get-column-values.ts](../samples/vault-api/get-column-values.ts) #### Redaction Types @@ -448,11 +450,11 @@ Use redaction types to control how sensitive data displays when retrieved from t - Use `MASKED` to provide partial visibility of sensitive data for less critical use cases. - Use `PLAIN_TEXT` for internal, authorized access where full data visibility is necessary. -For all `GetOptions` setters and `OrderByEnum` values, see [GetOptions](docs/api_reference.md#getoptions) in the API reference. +For all `GetOptions` setters and `OrderByEnum` values, see [GetOptions](../docs/api_reference.md#getoptions) in the API reference. ### Update Records -Update data in your vault using the `update` method. Create an update request with the [`UpdateRequest`](docs/api_reference.md#updaterequest) class, specifying parameters such as the table name and data (as a dictionary). +Update data in your vault using the `update` method. Create an update request with the [`UpdateRequest`](../docs/api_reference.md#updaterequest) class, specifying parameters such as the table name and data (as a dictionary). Configure update options using the `UpdateOptions` class to control returnTokens, tokens, and tokenMode. When `returnTokens` is `true`, Skyflow returns tokens for the updated records. When `false`, Skyflow returns IDs for the updated records. @@ -477,7 +479,7 @@ const response: UpdateResponse = await skyflowClient console.log('Update response:', response); ``` -Returns an [`UpdateResponse`](docs/api_reference.md#updateresponse) (`updatedField`, `errors`). With the default `setReturnTokens(false)`, `updatedField` contains only `skyflowId`: +Returns an [`UpdateResponse`](../docs/api_reference.md#updateresponse) (`updatedField`, `errors`). With the default `setReturnTokens(false)`, `updatedField` contains only `skyflowId`: ```text Update response: { updatedField: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c' }, errors: null } @@ -485,14 +487,14 @@ Update response: { updatedField: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5 > **Note:** The response key is `skyflowId`. The legacy `skyflow_id` key is deprecated and will be removed in an upcoming release. -For all `UpdateOptions` setters and `TokenMode` values, see [UpdateOptions](docs/api_reference.md#updateoptions) in the API reference. +For all `UpdateOptions` setters and `TokenMode` values, see [UpdateOptions](../docs/api_reference.md#updateoptions) in the API reference. > [!TIP] -> See the full example in the samples directory: [update-record.ts](samples/vault-api/update-record.ts) +> See the full example in the samples directory: [update-record.ts](../samples/vault-api/update-record.ts) ### Delete Records -Delete records using Skyflow IDs with the `delete` method. Create a delete request with the [`DeleteRequest`](docs/api_reference.md#deleterequest) class, which accepts a list of Skyflow IDs: +Delete records using Skyflow IDs with the `delete` method. Create a delete request with the [`DeleteRequest`](../docs/api_reference.md#deleterequest) class, which accepts a list of Skyflow IDs: ```typescript import { DeleteRequest, DeleteResponse } from "skyflow-node"; @@ -510,18 +512,18 @@ const response: DeleteResponse = await skyflowClient console.log("Delete response:", response); ``` -Returns a [`DeleteResponse`](docs/api_reference.md#deleteresponse) (`deletedIds`, `errors`): +Returns a [`DeleteResponse`](../docs/api_reference.md#deleteresponse) (`deletedIds`, `errors`): ```text Delete response: { deletedIds: ['a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c'], errors: null } ``` > [!TIP] -> See the full example in the samples directory: [delete-records.ts](samples/vault-api/delete-records.ts) +> See the full example in the samples directory: [delete-records.ts](../samples/vault-api/delete-records.ts) ### Query -Retrieve data with SQL queries using the `query` method. Create a query request with the [`QueryRequest`](docs/api_reference.md#queryrequest) class, which takes the `query` parameter as follows: +Retrieve data with SQL queries using the `query` method. Create a query request with the [`QueryRequest`](../docs/api_reference.md#queryrequest) class, which takes the `query` parameter as follows: @@ -539,20 +541,20 @@ const response: QueryResponse = await skyflowClient console.log("Query response:", response); ``` -Returns a [`QueryResponse`](docs/api_reference.md#queryresponse) (`fields`, `errors`). Each record also includes a `tokenizedData` map: +Returns a [`QueryResponse`](../docs/api_reference.md#queryresponse) (`fields`, `errors`). Each record also includes a `tokenizedData` map: ```text Query response: { fields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', card_number: '4111111111111112', tokenizedData: { card_number: '5391-4629-3722-7102' } }], errors: null } ``` > [!TIP] -> See the full example in the samples directory: [query-records.ts](samples/vault-api/query-records.ts) +> See the full example in the samples directory: [query-records.ts](../samples/vault-api/query-records.ts) Refer to [Query your data](https://docs.skyflow.com/query-data/) and [Execute Query](https://docs.skyflow.com/record/#QueryService_ExecuteQuery) for guidelines and restrictions on supported SQL statements, operators, and keywords. ### Upload File -Upload files to a Skyflow vault using the `uploadFile` method. Requires Node.js v20 or above. Create a [`FileUploadRequest`](docs/api_reference.md#fileuploadrequest) with the table name and column name, then configure a `FileUploadOptions` with the file source and optionally a Skyflow ID. +Upload files to a Skyflow vault using the `uploadFile` method. Requires Node.js v20 or above. Create a [`FileUploadRequest`](../docs/api_reference.md#fileuploadrequest) with the table name and column name, then configure a `FileUploadOptions` with the file source and optionally a Skyflow ID. **Attach a file to an existing record:** @@ -587,20 +589,20 @@ const response: FileUploadResponse = await skyflowClient console.log("File upload:", response); ``` -Both forms return a [`FileUploadResponse`](docs/api_reference.md#fileuploadresponse) (`skyflowId`, `errors`): +Both forms return a [`FileUploadResponse`](../docs/api_reference.md#fileuploadresponse) (`skyflowId`, `errors`): ```text File upload: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', errors: null } ``` -For all `FileUploadOptions` setters, see [FileUploadOptions](docs/api_reference.md#fileuploadoptions) in the API reference. +For all `FileUploadOptions` setters, see [FileUploadOptions](../docs/api_reference.md#fileuploadoptions) in the API reference. > [!TIP] -> See the full example in the samples directory: [file-upload.ts](samples/vault-api/file-upload.ts) +> See the full example in the samples directory: [file-upload.ts](../samples/vault-api/file-upload.ts) ### Retrieve Existing Tokens: `.tokenize(request)` -Retrieve tokens for values that already exist in the vault using the `.tokenize()` method. This method returns existing tokens only and does not generate new tokens. Build the request with the [`TokenizeRequest`](docs/api_reference.md#tokenizerequest) class. +Retrieve tokens for values that already exist in the vault using the `.tokenize()` method. This method returns existing tokens only and does not generate new tokens. Build the request with the [`TokenizeRequest`](../docs/api_reference.md#tokenizerequest) class. #### Construct a `.tokenize()` request @@ -619,14 +621,14 @@ const response: TokenizeResponse = await skyflowClient console.log("Tokenization Result:", response); ``` -Returns a [`TokenizeResponse`](docs/api_reference.md#tokenizeresponse) (`tokens`, `errors`): +Returns a [`TokenizeResponse`](../docs/api_reference.md#tokenizeresponse) (`tokens`, `errors`): ```text Tokenization Result: { tokens: ['5391-4629-3722-7102', '0f6b8a2c-90ab-4cde-9def-567890abcdef'], errors: null } ``` > [!TIP] -> See the full example in the samples directory: [tokenize-records.ts](samples/vault-api/tokenize-records.ts) +> See the full example in the samples directory: [tokenize-records.ts](../samples/vault-api/tokenize-records.ts) ## Detect @@ -636,19 +638,19 @@ Detect configurations are created in Studio or through the management API. The S | Method | Request | Response | Description | |--------|---------|----------|-------------| -| `deidentifyString(request)` | [`DeidentifyStringRequest`](docs/api_reference.md#deidentifystringrequest) | [`DeidentifyStringResponse`](docs/api_reference.md#deidentifystringresponse) | Detect and replace sensitive data in a string | -| `deidentifyFile(request)` | [`DeidentifyFileRequest`](docs/api_reference.md#deidentifyfilerequest) | [`DeidentifyFileResponse`](docs/api_reference.md#deidentifyfileresponse) | Submit a file for de-identification, optionally polling the run to completion | -| `reidentifyString(request)` | [`ReidentifyStringRequest`](docs/api_reference.md#reidentifystringrequest) | [`ReidentifyStringResponse`](docs/api_reference.md#reidentifystringresponse) | Replace tokens in a string with their original or redacted values | -| `reidentifyFile(request)` | [`ReidentifyFileRequest`](docs/api_reference.md#reidentifyfilerequest) | [`ReidentifyFileResponse`](docs/api_reference.md#reidentifyfileresponse) | Replace tokens in a file with their original or redacted values | -| `getRun(request)` | [`GetRunRequest`](docs/api_reference.md#getrunrequest) | [`GetRunResponse`](docs/api_reference.md#getrunresponse) | Retrieve the status and outputs of a Detect file run | -| `checkGuardrails(request)` | [`CheckGuardrailsRequest`](docs/api_reference.md#checkguardrailsrequest) | [`CheckGuardrailsResponse`](docs/api_reference.md#checkguardrailsresponse) | Screen text for toxicity and denied topics | +| `deidentifyString(request)` | [`DeidentifyStringRequest`](../docs/api_reference.md#deidentifystringrequest) | [`DeidentifyStringResponse`](../docs/api_reference.md#deidentifystringresponse) | Detect and replace sensitive data in a string | +| `deidentifyFile(request)` | [`DeidentifyFileRequest`](../docs/api_reference.md#deidentifyfilerequest) | [`DeidentifyFileResponse`](../docs/api_reference.md#deidentifyfileresponse) | Submit a file for de-identification, optionally polling the run to completion | +| `reidentifyString(request)` | [`ReidentifyStringRequest`](../docs/api_reference.md#reidentifystringrequest) | [`ReidentifyStringResponse`](../docs/api_reference.md#reidentifystringresponse) | Replace tokens in a string with their original or redacted values | +| `reidentifyFile(request)` | [`ReidentifyFileRequest`](../docs/api_reference.md#reidentifyfilerequest) | [`ReidentifyFileResponse`](../docs/api_reference.md#reidentifyfileresponse) | Replace tokens in a file with their original or redacted values | +| `getRun(request)` | [`GetRunRequest`](../docs/api_reference.md#getrunrequest) | [`GetRunResponse`](../docs/api_reference.md#getrunresponse) | Retrieve the status and outputs of a Detect file run | +| `checkGuardrails(request)` | [`CheckGuardrailsRequest`](../docs/api_reference.md#checkguardrailsrequest) | [`CheckGuardrailsResponse`](../docs/api_reference.md#checkguardrailsresponse) | Screen text for toxicity and denied topics | **Before you start** - **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy that grants the Detokenization action on the destination table and columns, unless it is a Vault Owner. - **Configuration.** `configurationId` is the id of a Detect configuration created through the management API. The alternative is an inline `DetectConfiguration` passed as `configuration` (see [Detect configuration](#detect-configuration)). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings. Sending both is rejected by the API with HTTP 400. - **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the `table.column` named by `destination`. That table and column must already exist in the vault. -- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](docs/api_reference.md#detect-api). +- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](../docs/api_reference.md#detect-api). ### De-identify a string: `.deidentifyString(request)` @@ -703,7 +705,7 @@ console.log(inlineResponse.metrics?.wordCount); `DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. > [!TIP] -> See the full example in the samples directory: [deidentify-string.ts](samples/detect-api/deidentify-string.ts) +> See the full example in the samples directory: [deidentify-string.ts](../samples/detect-api/deidentify-string.ts) ### Detect configuration @@ -712,9 +714,9 @@ console.log(inlineResponse.metrics?.wordCount); | Type | Fields | Notes | |------|--------|-------| | `Detect` | `entities`, `objectEntities`, `restrict`, `skip`, `customTypes`, `returnEntities` | `restrict` and `skip` are regexes that are always, respectively never, treated as sensitive. `returnEntities` is `EXCLUDE_SENSITIVE_DATA` (server default), `ALL` or `NONE`. | -| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination` is used only with `VAULT_TOKEN`. | +| `Entity` | `entityType`, `deidentificationType`, `destination`, `transformation`, `includePrefix`, `prefixOverride` | `entityType` defaults to `ALL`. `deidentificationType` is `ENTITY_UNIQUE_COUNTER` (server default), `ENTITY_ONLY`, `MASK` or `VAULT_TOKEN`. `destination`, `includePrefix` and `prefixOverride` apply only with `VAULT_TOKEN`: `includePrefix` is `YES` (server default), `NO` (raw token, cannot be re-identified) or `INCLUDE_PREFIX_UNSPECIFIED` (field omitted); `prefixOverride` replaces the entity-type prefix in the token and is not valid with `includePrefix` `NO`. | | `Transformation` / `ShiftDates` | `shiftDates`; `minDays`, `maxDays` | Shifts detected dates by a random number of days within the range. | -| `CustomType` | `label`, `match` | A regex-defined entity type. `label` is reported as the entity type of its matches. | +| `CustomType` | `label`, `match`, `deidentificationType`, `destination`, `includePrefix`, `prefixOverride` | A regex-defined entity type. `label` is reported as the entity type of its matches and is the default token prefix. The remaining fields work as on `Entity`; `destination` is required with `VAULT_TOKEN`. | | `ObjectEntity` | `entityType`, `deidentificationType` | Visual objects in images: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE` or `ALL`, handled with `REDACT` or `UNREDACT`. File inputs only. | | `FileMapping` | `source`, `destination`, `outputFileNameSuffix`, `entitiesDestination`, `objectEntitiesDestination` | Where the processed file and the entity files are written. File inputs only. | | `Media` | `audio`, `document`, `image` | Media-type options. File inputs only. | @@ -812,7 +814,7 @@ switch (response.status) { Outputs are written only when polling reached `SUCCESS`, the request was built from a local file, and the run delivered them as base64. A request built from a `SKYFLOW_ID` or `PRESIGNED_URL` has no local name to derive from, and presigned-URL outputs are not fetched, so in those cases read `output` and write the files yourself. Files are named `processed-.`, with the entities output as `processed-entities-.json`; `outputDirectory` is created if missing and defaults to the working directory. Writing outputs needs the Node.js file system and is not supported in Cloudflare Workers. > [!TIP] -> See the full examples in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](samples/detect-api/deidentify-file-async.ts) +> See the full examples in the samples directory: [deidentify-file.ts](../samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](../samples/detect-api/deidentify-file-async.ts) ### Re-identify a string: `.reidentifyString(request)` @@ -835,7 +837,7 @@ console.log(response.processedText); `RedactionLevel` is the skyvault SDK's own type and has only `entityName` and `redactionType`; the `tokenGroupName` and `redactionPattern` rules of the flowvault SDK do not exist here. Both fields are optional on the API, so the SDK passes the list through as given and the server rejects invalid combinations. > [!TIP] -> See the full example in the samples directory: [reidentify-string.ts](samples/detect-api/reidentify-string.ts) +> See the full example in the samples directory: [reidentify-string.ts](../samples/detect-api/reidentify-string.ts) ### Re-identify a file: `.reidentifyFile(request)` @@ -862,7 +864,7 @@ if (response.status === DetectRunStatus.SUCCESS) { `ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. > [!TIP] -> See the full example in the samples directory: [reidentify-file.ts](samples/detect-api/reidentify-file.ts) +> See the full example in the samples directory: [reidentify-file.ts](../samples/detect-api/reidentify-file.ts) ### Get run: `.getRun(request)` @@ -890,7 +892,7 @@ switch (run.status) { ``` > [!TIP] -> See the full example in the samples directory: [get-run.ts](samples/detect-api/get-run.ts) +> See the full example in the samples directory: [get-run.ts](../samples/detect-api/get-run.ts) ### Check guardrails: `.checkGuardrails(request)` @@ -911,7 +913,7 @@ if (response.validation === GuardrailsValidation.FAILED) { All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling). Client-side validation fails before any network call. > [!TIP] -> See the full example in the samples directory: [check-guardrails.ts](samples/detect-api/check-guardrails.ts) +> See the full example in the samples directory: [check-guardrails.ts](../samples/detect-api/check-guardrails.ts) ## Connections @@ -936,7 +938,7 @@ const connectionConfig: ConnectionConfig = { ### Invoke a connection -Invoke a connection using the `invoke` method of the Skyflow client. Build the request with the [`InvokeConnectionRequest`](docs/api_reference.md#invokeconnectionrequest) class. +Invoke a connection using the `invoke` method of the Skyflow client. Build the request with the [`InvokeConnectionRequest`](../docs/api_reference.md#invokeconnectionrequest) class. #### Construct an invoke connection request @@ -962,13 +964,13 @@ const response: InvokeConnectionResponse = await skyflowClient console.log("Invoke connection response:", response); ``` -Returns an [`InvokeConnectionResponse`](docs/api_reference.md#invokeconnectionresponse) (`data`, `metadata`, `errors`): +Returns an [`InvokeConnectionResponse`](../docs/api_reference.md#invokeconnectionresponse) (`data`, `metadata`, `errors`): ```text Invoke connection response: { data: { status: 'success', transaction_id: 'txn_abc123' }, metadata: { requestId: 'req-xyz-456' }, errors: null } ``` -`method` supports the following methods (see [`RequestMethod`](docs/api_reference.md#requestmethod)): +`method` supports the following methods (see [`RequestMethod`](../docs/api_reference.md#requestmethod)): - `GET` - `POST` @@ -978,7 +980,7 @@ Invoke connection response: { data: { status: 'success', transaction_id: 'txn_ab **pathParams, queryParams, header, body** are the JSON objects represented as dictionaries that will be sent through the connection integration url. > [!TIP] -> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/vault-api/invoke-connection.ts) +> See the full example in the samples directory: [scoped-token-generation-example.ts](../samples/vault-api/invoke-connection.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on integrations with Connections, Functions, and Pipelines. ## Authentication & authorization @@ -1110,7 +1112,7 @@ const options: BearerTokenOptions = { ``` > [!TIP] -> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/service-account/scoped-token-generation-example.ts) +> See the full example in the samples directory: [scoped-token-generation-example.ts](../samples/service-account/scoped-token-generation-example.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. #### Generate bearer tokens with `ctx` for context-aware authorization @@ -1165,7 +1167,7 @@ const credentials: PathCredentials = { ``` > [!TIP] -> See the full example in the samples directory: [token-generation-with-context-example.ts](samples/service-account/token-generation-with-context-example.ts) +> See the full example in the samples directory: [token-generation-with-context-example.ts](../samples/service-account/token-generation-with-context-example.ts) > See Skyflow's [context-aware authorization](https://docs.skyflow.com) and [conditional data access](https://docs.skyflow.com) docs for policy variable syntax like `request.context.*`. #### Generate signed data tokens @@ -1222,12 +1224,12 @@ type SignedDataTokensOptions = { ``` > [!TIP] -> See the full example in the samples directory: [signed-token-generation-example.ts](samples/service-account/signed-token-generation-example.ts) +> See the full example in the samples directory: [signed-token-generation-example.ts](../samples/service-account/signed-token-generation-example.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. ## Client management -After initializing the `Skyflow` client, you can add, update, retrieve, and remove vault and connection configurations at any time without recreating the client. For the full list of methods, see [Client management methods](docs/api_reference.md#client-management-methods) in the API reference. +After initializing the `Skyflow` client, you can add, update, retrieve, and remove vault and connection configurations at any time without recreating the client. For the full list of methods, see [Client management methods](../docs/api_reference.md#client-management-methods) in the API reference. ## Using the client in production @@ -1347,7 +1349,7 @@ message: Authentication failed. Bearer token is expired. Use a valid bearer toke If you encounter this kind of error, retry the request. During the retry the SDK detects that the previous bearer token has expired and generates a new one for the current and subsequent requests. > [!TIP] -> See the full example in the samples directory: [bearer-token-expiry-example.ts](samples/service-account/bearer-token-expiry-example.ts) +> See the full example in the samples directory: [bearer-token-expiry-example.ts](../samples/service-account/bearer-token-expiry-example.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. ## Troubleshooting @@ -1370,7 +1372,7 @@ If you're stuck, set `logLevel: LogLevel.DEBUG` during development for detailed ## TypeScript types reference -For the full reference of all exported types, interfaces, enums, request/response fields, and options setters, see the **[API Reference](docs/api_reference.md)**. +For the full reference of all exported types, interfaces, enums, request/response fields, and options setters, see the **[API Reference](../docs/api_reference.md)**. The most commonly imported TypeScript types are: diff --git a/skyvault/api-report/skyflow-node.api.md b/skyvault/api-report/skyflow-node.api.md index 39b00709..e068df04 100644 --- a/skyvault/api-report/skyflow-node.api.md +++ b/skyvault/api-report/skyflow-node.api.md @@ -5,9 +5,11 @@ ```ts import { ApiKeyCredentials } from 'skyflow-common'; +import { Audio as Audio_2 } from 'skyflow-common'; import { AuthInfo } from 'skyflow-common'; import { BaseDeleteRequest } from 'skyflow-common'; import { BaseDeleteResponse } from 'skyflow-common'; +import { BaseDetectController } from 'skyflow-common'; import { BaseDetokenizeRequest } from 'skyflow-common'; import { BaseDetokenizeResponse } from 'skyflow-common'; import { BaseFileUploadRequest } from 'skyflow-common'; @@ -28,13 +30,34 @@ import { BaseUpdateRequest } from 'skyflow-common'; import { BaseUpdateResponse } from 'skyflow-common'; import { BaseVaultEntry } from 'skyflow-common'; import { BearerTokenOptions } from 'skyflow-common'; +import { Bleep } from 'skyflow-common'; import { Blob as Blob_2 } from 'buffer'; +import { CheckGuardrailsRequest } from 'skyflow-common'; +import { CheckGuardrailsRequestFields } from 'skyflow-common'; +import { CheckGuardrailsResponse } from 'skyflow-common'; import { ConnectionConfig } from 'skyflow-common'; import { Credentials } from 'skyflow-common'; -import { DetectEntities } from 'skyflow-common'; -import { DetectOutputTranscription } from 'skyflow-common'; +import { CustomType } from 'skyflow-common'; +import { DeidentificationType } from 'skyflow-common'; +import { DeidentifyFileRequest } from 'skyflow-common'; +import { DeidentifyFileRequestFields } from 'skyflow-common'; +import { DeidentifyFileResponse } from 'skyflow-common'; +import { DeidentifyStringRequest } from 'skyflow-common'; +import { DeidentifyStringRequestFields } from 'skyflow-common'; +import { DeidentifyStringResponse } from 'skyflow-common'; +import { Detect } from 'skyflow-common'; +import { DetectConfiguration } from 'skyflow-common'; +import { DetectedEntity } from 'skyflow-common'; +import { DetectRunFields } from 'skyflow-common'; +import { DetectRunStatus } from 'skyflow-common'; +import { Document as Document_2 } from 'skyflow-common'; +import { Entity } from 'skyflow-common'; +import { EntityLocation } from 'skyflow-common'; +import { EntityType } from 'skyflow-common'; import { Env } from 'skyflow-common'; -import { FileInput } from 'skyflow-common'; +import { FileDataFormat } from 'skyflow-common'; +import { FileMapping } from 'skyflow-common'; +import { FileOutput } from 'skyflow-common'; import * as fs from 'fs'; import { generateBearerToken } from 'skyflow-common'; import { generateBearerTokenFromCreds } from 'skyflow-common'; @@ -42,16 +65,39 @@ import { generateSignedDataTokens } from 'skyflow-common'; import { generateSignedDataTokensFromCreds } from 'skyflow-common'; import { GenerateTokenOptions } from 'skyflow-common'; import { GetResponseData } from 'skyflow-common'; -import { IndexRange } from 'skyflow-common'; +import { GetRunRequest } from 'skyflow-common'; +import { GetRunResponse } from 'skyflow-common'; +import { GuardrailsValidation } from 'skyflow-common'; +import { Image as Image_2 } from 'skyflow-common'; +import { IncludePrefix } from 'skyflow-common'; import { InsertResponseType } from 'skyflow-common'; import { isExpired } from 'skyflow-common'; import { LogLevel } from 'skyflow-common'; import { MaskingMethod } from 'skyflow-common'; +import { Media } from 'skyflow-common'; +import { Metrics } from 'skyflow-common'; +import { ObjectDeidentificationType } from 'skyflow-common'; +import { ObjectEntity } from 'skyflow-common'; +import { ObjectEntityType } from 'skyflow-common'; import { OrderByEnum } from 'skyflow-common'; +import { OutputTranscriptionType } from 'skyflow-common'; import { PathCredentials } from 'skyflow-common'; +import { Pdf } from 'skyflow-common'; +import { PdfProcessingMode } from 'skyflow-common'; +import { POLL_OPTIONS } from 'skyflow-common'; +import { PollOptions } from 'skyflow-common'; import { QueryResponseType } from 'skyflow-common'; +import { RedactionRule } from 'skyflow-common'; import { RedactionType } from 'skyflow-common'; +import { ReidentifyFileRequest } from 'skyflow-common'; +import { ReidentifyFileRequestFields } from 'skyflow-common'; +import { ReidentifyFileResponse } from 'skyflow-common'; +import { ReidentifyStringRequest } from 'skyflow-common'; +import { ReidentifyStringRequestFields } from 'skyflow-common'; +import { ReidentifyStringResponse } from 'skyflow-common'; import { RequestMethod } from 'skyflow-common'; +import { ReturnEntitiesType } from 'skyflow-common'; +import { ShiftDates } from 'skyflow-common'; import { SignedDataTokensOptions } from 'skyflow-common'; import { SkyflowConfig } from 'skyflow-common'; import { SkyflowError } from 'skyflow-common'; @@ -61,232 +107,53 @@ import { StringKeyValueMapType } from 'skyflow-common'; import { TokenCredentials } from 'skyflow-common'; import { TokenizeRequestType } from 'skyflow-common'; import { TokenMode } from 'skyflow-common'; -import { TokenType } from 'skyflow-common'; +import { Transformation } from 'skyflow-common'; import { VaultConfig } from 'skyflow-common'; export { ApiKeyCredentials } +export { Audio_2 as Audio } + export { BearerTokenOptions } -// @public (undocumented) -export class Bleep { - // (undocumented) - getFrequency(): number | undefined; - // (undocumented) - getGain(): number | undefined; - // (undocumented) - getStartPadding(): number | undefined; - // (undocumented) - getStopPadding(): number | undefined; - // (undocumented) - setFrequency(frequency: number): void; - // (undocumented) - setGain(gain: number): void; - // (undocumented) - setStartPadding(startPadding: number): void; - // (undocumented) - setStopPadding(stopPadding: number): void; -} +export { Bleep } + +export { CheckGuardrailsRequest } + +export { CheckGuardrailsRequestFields } + +export { CheckGuardrailsResponse } export { ConnectionConfig } export { Credentials } -// @public (undocumented) -export class DeidentifyFileOptions { - constructor(); - // (undocumented) - getAllowRegexList(): string[] | undefined; - // (undocumented) - getBleep(): Bleep | undefined; - // (undocumented) - getEntities(): string[] | undefined; - // (undocumented) - getMaskingMethod(): MaskingMethod | undefined; - // (undocumented) - getMaxResolution(): number | undefined; - // (undocumented) - getOutputDirectory(): string | undefined; - // (undocumented) - getOutputOcrText(): boolean | undefined; - // (undocumented) - getOutputProcessedAudio(): boolean | undefined; - // (undocumented) - getOutputProcessedImage(): boolean | undefined; - // (undocumented) - getOutputTranscription(): DetectOutputTranscription | undefined; - // (undocumented) - getPixelDensity(): number | undefined; - // (undocumented) - getRestrictRegexList(): string[] | undefined; - // (undocumented) - getTokenFormat(): TokenFormat | undefined; - // (undocumented) - getTransformations(): Transformations | undefined; - // (undocumented) - getWaitTime(): number | undefined; - // (undocumented) - setAllowRegexList(allowRegexList: string[]): void; - // (undocumented) - setBleep(bleep: Bleep | undefined): void; - // (undocumented) - setEntities(entities: DetectEntities[]): void; - // (undocumented) - setMaskingMethod(maskingMethod: MaskingMethod | undefined): void; - // (undocumented) - setMaxResolution(maxResolution: number | undefined): void; - // (undocumented) - setOutputDirectory(outputDirectory: string | undefined): void; - // (undocumented) - setOutputOcrText(outputOcrText: boolean | undefined): void; - // (undocumented) - setOutputProcessedAudio(outputProcessedAudio: boolean | undefined): void; - // (undocumented) - setOutputProcessedImage(value: boolean | undefined): void; - // (undocumented) - setOutputTranscription(outputTranscription: DetectOutputTranscription | undefined): void; - // (undocumented) - setPixelDensity(pixelDensity: number | undefined): void; - // (undocumented) - setRestrictRegexList(restrictRegexList: string[]): void; - // (undocumented) - setTokenFormat(tokenFormat: TokenFormat): void; - // (undocumented) - setTransformations(transformations: Transformations): void; - // (undocumented) - setWaitTime(waitTime: number | undefined): void; -} +export { CustomType } -// @public (undocumented) -export class DeidentifyFileRequest { - constructor(file: FileInput); - // (undocumented) - getFile(): FileInput; - // (undocumented) - setFile(file: FileInput): void; -} +export { DeidentificationType } -// @public (undocumented) -export class DeidentifyFileResponse { - constructor(input: { - fileBase64?: string; - file?: File; - type?: string; - extension?: string; - wordCount?: number; - charCount?: number; - sizeInKb?: number; - durationInSeconds?: number; - pageCount?: number; - slideCount?: number; - entities?: Array<{ - file: string; - extension: string; - }>; - runId?: string; - status?: string; - errors?: Array | null; - }); - // (undocumented) - charCount?: number; - // (undocumented) - durationInSeconds?: number; - // (undocumented) - entities?: Array<{ - file: string; - extension: string; - }>; - // (undocumented) - errors: Array | null; - // (undocumented) - extension?: string; - // (undocumented) - file?: File; - // (undocumented) - fileBase64?: string; - // (undocumented) - pageCount?: number; - // (undocumented) - runId?: string; - // (undocumented) - sizeInKb?: number; - // (undocumented) - slideCount?: number; - // (undocumented) - status?: string; - // (undocumented) - type?: string; - // (undocumented) - wordCount?: number; -} +export { DeidentifyFileRequest } -// @public (undocumented) -export class DeidentifyTextOptions { - // (undocumented) - getAllowRegexList(): string[] | undefined; - // (undocumented) - getEntities(): DetectEntities[] | undefined; - // (undocumented) - getRestrictRegexList(): string[] | undefined; - // (undocumented) - getTokenFormat(): TokenFormat | undefined; - // (undocumented) - getTransformations(): Transformations | undefined; - // (undocumented) - setAllowRegexList(allowRegexList: string[]): void; - // (undocumented) - setEntities(entities: DetectEntities[]): void; - // (undocumented) - setRestrictRegexList(restrictRegexList: string[]): void; - // (undocumented) - setTokenFormat(tokenFormat: TokenFormat): void; - // (undocumented) - setTransformations(transformations: Transformations): void; -} +export { DeidentifyFileRequestFields } -// @public (undocumented) -export class DeidentifyTextRequest { - constructor(text: string); - // (undocumented) - get text(): string; - set text(value: string); -} +export { DeidentifyFileResponse } -// @public (undocumented) -export class DeidentifyTextResponse { - constructor(input: { - processedText: string; - entities: Array<{ - token?: string; - value?: string; - textIndex?: IndexRange; - processedIndex?: IndexRange; - entity?: string; - scores?: Record; - }>; - wordCount: number; - charCount: number; - errors?: Array | null; - }); - // (undocumented) - charCount: number; - // (undocumented) - entities: Array<{ - token?: string; - value?: string; - textIndex?: IndexRange; - processedIndex?: IndexRange; - entity?: string; - scores?: Record; - }>; +// @public +export enum DeidentifySourceType { // (undocumented) - errors: Array | null; + BASE64 = "BASE64", // (undocumented) - processedText: string; + PRESIGNED_URL = "PRESIGNED_URL", // (undocumented) - wordCount: number; + SKYFLOW_ID = "SKYFLOW_ID" } +export { DeidentifyStringRequest } + +export { DeidentifyStringRequestFields } + +export { DeidentifyStringResponse } + // @public (undocumented) export class DeleteRequest extends BaseDeleteRequest { constructor(table: string, deleteIds: Array); @@ -311,9 +178,15 @@ export class DeleteResponse extends BaseDeleteResponse { errors: Array | null; } -export { DetectEntities } +export { Detect } + +export { DetectConfiguration } + +export { DetectedEntity } -export { DetectOutputTranscription } +export { DetectRunFields } + +export { DetectRunStatus } // @public (undocumented) export interface DetokenizeData { @@ -362,9 +235,21 @@ export class DetokenizeResponse extends BaseDetokenizeResponse { errors: Array | null; } +export { Document_2 as Document } + +export { Entity } + +export { EntityLocation } + +export { EntityType } + export { Env } -export { FileInput } +export { FileDataFormat } + +export { FileMapping } + +export { FileOutput } // @public (undocumented) export class FileUploadOptions extends BaseOptions { @@ -441,14 +326,6 @@ export class GetColumnRequest { set table(value: string); } -// @public (undocumented) -export class GetDetectRunRequest { - constructor(runId: string); - // (undocumented) - get runId(): string; - set runId(value: string); -} - // @public (undocumented) export class GetOptions extends BaseOptions { constructor(); @@ -519,7 +396,15 @@ export class GetResponse extends BaseGetResponse { export { GetResponseData } -export { IndexRange } +export { GetRunRequest } + +export { GetRunResponse } + +export { GuardrailsValidation } + +export { Image_2 as Image } + +export { IncludePrefix } // @public (undocumented) export class InsertOptions extends BaseOptions { @@ -612,10 +497,30 @@ export { LogLevel } export { MaskingMethod } +export { Media } + +export { Metrics } + +export { ObjectDeidentificationType } + +export { ObjectEntity } + +export { ObjectEntityType } + export { OrderByEnum } +export { OutputTranscriptionType } + export { PathCredentials } +export { Pdf } + +export { PdfProcessingMode } + +export { POLL_OPTIONS } + +export { PollOptions } + // @public (undocumented) export class QueryRequest extends BaseQueryRequest { constructor(query: string); @@ -638,43 +543,44 @@ export class QueryResponse extends BaseQueryResponse { export { QueryResponseType } -export { RedactionType } - -// @public (undocumented) -export class ReidentifyTextOptions { - // (undocumented) - getMaskedEntities(): DetectEntities[] | undefined; - // (undocumented) - getPlainTextEntities(): DetectEntities[] | undefined; - // (undocumented) - getRedactedEntities(): DetectEntities[] | undefined; - // (undocumented) - setMaskedEntities(maskedEntities: DetectEntities[]): void; +// @public +export interface RedactionLevel extends RedactionRule { + entityName?: EntityType; // (undocumented) - setPlainTextEntities(plainTextEntities: DetectEntities[]): void; + redactionPattern?: never; + redactionType?: RedactionType; // (undocumented) - setRedactedEntities(redactedEntities: DetectEntities[]): void; + tokenGroupName?: never; } -// @public (undocumented) -export class ReidentifyTextRequest { - constructor(text: string); - // (undocumented) - get text(): string; - set text(value: string); -} +export { RedactionType } -// @public (undocumented) -export class ReidentifyTextResponse { - constructor(input: { - processedText: string; - }); +export { ReidentifyFileRequest } + +export { ReidentifyFileRequestFields } + +export { ReidentifyFileResponse } + +// @public +export enum ReidentifySourceType { // (undocumented) - processedText: string; + BASE64 = "BASE64", + // (undocumented) + PRESIGNED_URL = "PRESIGNED_URL" } +export { ReidentifyStringRequest } + +export { ReidentifyStringRequestFields } + +export { ReidentifyStringResponse } + export { RequestMethod } +export { ReturnEntitiesType } + +export { ShiftDates } + export { SignedDataTokensOptions } // Warning: (ae-forgotten-export) The symbol "VaultEntry" needs to be exported by the entry point index.d.ts @@ -722,26 +628,6 @@ export { StringKeyValueMapType } export { TokenCredentials } -// @public (undocumented) -export class TokenFormat { - // (undocumented) - getDefault(): string | undefined; - // (undocumented) - getEntityOnly(): string[] | undefined; - // (undocumented) - getEntityUniqueCounter(): string[] | undefined; - // (undocumented) - getVaultToken(): string[] | undefined; - // (undocumented) - setDefault(defaultToken: TokenType): void; - // (undocumented) - setEntityOnly(entityOnly: DetectEntities[]): void; - // (undocumented) - setEntityUniqueCounter(entityUniqueCounter: DetectEntities[]): void; - // (undocumented) - setVaultToken(vaultToken: DetectEntities[]): void; -} - // @public (undocumented) export class TokenizeRequest extends BaseTokenizeRequest { constructor(values: Array); @@ -766,23 +652,7 @@ export class TokenizeResponse extends BaseTokenizeResponse { export { TokenMode } -export { TokenType } - -// @public (undocumented) -export class Transformations { - // (undocumented) - getShiftDays(): { - max: number; - min: number; - entities: DetectEntities[]; - } | undefined; - // (undocumented) - setShiftDays(shiftDays: { - max: number; - min: number; - entities: DetectEntities[]; - }): void; -} +export { Transformation } // @public (undocumented) export class UpdateOptions extends BaseOptions { diff --git a/skyvault/src/index.ts b/skyvault/src/index.ts index 7dc73ee8..c9590c28 100644 --- a/skyvault/src/index.ts +++ b/skyvault/src/index.ts @@ -48,6 +48,7 @@ import { CheckGuardrailsResponse, EntityType, DeidentificationType, + IncludePrefix, FileDataFormat, DetectRunStatus, ReturnEntitiesType, @@ -183,6 +184,7 @@ export { FileOutput, EntityType, DeidentificationType, + IncludePrefix, DeidentifySourceType, ReidentifySourceType, FileDataFormat, From a8920c2e3156d92c7d6e9554548b7c0808226bce Mon Sep 17 00:00:00 2001 From: skyflow-bharti Date: Tue, 10 Nov 2026 09:29:54 +0530 Subject: [PATCH 09/11] SK-3076 updated samples inside skyvault --- AGENTS.md | 4 +- README.md | 7 +- docs/api_reference.md | 4 +- docs/migrate_detect.md | 399 ++++++++++++++++++ flowvault/README.md | 185 +++++++- .../samples/detect-api/reidentify-string.ts | 60 ++- samples/detect-api/reidentify-string.ts | 86 ---- skyvault/README.md | 247 +++++++++-- {samples => skyvault/samples}/README.md | 2 +- .../bearer-token-expiry-example.ts | 0 .../scoped-token-generation-example.ts | 0 .../vault-api/credentials-options.ts | 0 .../vault-api/detokenize-records.ts | 0 .../deprecated/vault-api/file-upload.ts | 0 .../deprecated/vault-api/get-records.ts | 0 .../deprecated/vault-api/update-record.ts | 0 .../samples}/detect-api/check-guardrails.ts | 0 .../detect-api/deidentify-file-async.ts | 0 .../samples}/detect-api/deidentify-file.ts | 0 .../samples}/detect-api/deidentify-string.ts | 0 .../samples}/detect-api/get-run.ts | 0 .../samples}/detect-api/reidentify-file.ts | 0 .../samples/detect-api/reidentify-string.ts | 122 ++++++ .../samples}/package-lock.json | 0 {samples => skyvault/samples}/package.json | 0 .../bearer-token-expiry-example.ts | 0 .../scoped-token-generation-example.ts | 0 .../signed-token-generation-example.ts | 0 .../token-generation-example.ts | 0 .../token-generation-with-context-example.ts | 0 {samples => skyvault/samples}/vault-api/.env | 0 .../samples}/vault-api/client-operations.ts | 0 .../samples}/vault-api/credentials-options.ts | 0 .../samples}/vault-api/data-residency.ts | 0 .../samples}/vault-api/delete-records.ts | 0 .../samples}/vault-api/detokenzie-records.ts | 0 .../samples}/vault-api/file-upload.ts | 0 .../samples}/vault-api/get-column-values.ts | 0 .../samples}/vault-api/get-records.ts | 0 .../samples}/vault-api/insert-byot.ts | 0 .../vault-api/insert-continue-on-error.ts | 0 .../samples}/vault-api/insert-records.ts | 0 .../samples}/vault-api/invoke-connection.ts | 0 .../samples}/vault-api/query-records.ts | 0 .../samples}/vault-api/tokenize-records.ts | 0 .../samples}/vault-api/update-record.ts | 0 46 files changed, 955 insertions(+), 161 deletions(-) create mode 100644 docs/migrate_detect.md delete mode 100644 samples/detect-api/reidentify-string.ts rename {samples => skyvault/samples}/README.md (99%) rename {samples => skyvault/samples}/deprecated/service-account/bearer-token-expiry-example.ts (100%) rename {samples => skyvault/samples}/deprecated/service-account/scoped-token-generation-example.ts (100%) rename {samples => skyvault/samples}/deprecated/vault-api/credentials-options.ts (100%) rename {samples => skyvault/samples}/deprecated/vault-api/detokenize-records.ts (100%) rename {samples => skyvault/samples}/deprecated/vault-api/file-upload.ts (100%) rename {samples => skyvault/samples}/deprecated/vault-api/get-records.ts (100%) rename {samples => skyvault/samples}/deprecated/vault-api/update-record.ts (100%) rename {samples => skyvault/samples}/detect-api/check-guardrails.ts (100%) rename {samples => skyvault/samples}/detect-api/deidentify-file-async.ts (100%) rename {samples => skyvault/samples}/detect-api/deidentify-file.ts (100%) rename {samples => skyvault/samples}/detect-api/deidentify-string.ts (100%) rename {samples => skyvault/samples}/detect-api/get-run.ts (100%) rename {samples => skyvault/samples}/detect-api/reidentify-file.ts (100%) create mode 100644 skyvault/samples/detect-api/reidentify-string.ts rename {samples => skyvault/samples}/package-lock.json (100%) rename {samples => skyvault/samples}/package.json (100%) rename {samples => skyvault/samples}/service-account/bearer-token-expiry-example.ts (100%) rename {samples => skyvault/samples}/service-account/scoped-token-generation-example.ts (100%) rename {samples => skyvault/samples}/service-account/signed-token-generation-example.ts (100%) rename {samples => skyvault/samples}/service-account/token-generation-example.ts (100%) rename {samples => skyvault/samples}/service-account/token-generation-with-context-example.ts (100%) rename {samples => skyvault/samples}/vault-api/.env (100%) rename {samples => skyvault/samples}/vault-api/client-operations.ts (100%) rename {samples => skyvault/samples}/vault-api/credentials-options.ts (100%) rename {samples => skyvault/samples}/vault-api/data-residency.ts (100%) rename {samples => skyvault/samples}/vault-api/delete-records.ts (100%) rename {samples => skyvault/samples}/vault-api/detokenzie-records.ts (100%) rename {samples => skyvault/samples}/vault-api/file-upload.ts (100%) rename {samples => skyvault/samples}/vault-api/get-column-values.ts (100%) rename {samples => skyvault/samples}/vault-api/get-records.ts (100%) rename {samples => skyvault/samples}/vault-api/insert-byot.ts (100%) rename {samples => skyvault/samples}/vault-api/insert-continue-on-error.ts (100%) rename {samples => skyvault/samples}/vault-api/insert-records.ts (100%) rename {samples => skyvault/samples}/vault-api/invoke-connection.ts (100%) rename {samples => skyvault/samples}/vault-api/query-records.ts (100%) rename {samples => skyvault/samples}/vault-api/tokenize-records.ts (100%) rename {samples => skyvault/samples}/vault-api/update-record.ts (100%) diff --git a/AGENTS.md b/AGENTS.md index 0b0128d6..29b10d21 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -163,7 +163,7 @@ All inputs are validated using functions in `src/utils/validations/index.ts`: - **Type definitions**: `src/vault/types/index.ts` - **Utilities**: `src/utils/` (validations, logging, JWT helpers) - **Service account auth**: `src/service-account/` -- **Sample code**: `samples/` (use as reference for common patterns) +- **Sample code**: `skyvault/samples/` and `flowvault/samples/` (use as reference for common patterns) ## Special Considerations @@ -193,7 +193,7 @@ All inputs are validated using functions in `src/utils/validations/index.ts`: 3. Update `src/index.ts` with exports 4. Add validation in `src/utils/validations/` 5. Write tests in `test/` -6. Add sample in `samples/` directory +6. Add sample in `skyvault/samples/` (or `flowvault/samples/`) directory 7. Update TypeDoc comments for documentation generation ## Documentation & README Standards diff --git a/README.md b/README.md index 3dc3a871..183f3f6e 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,9 @@ This repository hosts Skyflow's Node.js SDKs for integrating Skyflow into a Node > **The two packages are versioned independently.** `flowvault` is a new SDK starting at `1.0.0`; its lower version number reflects a first release, not an older or lesser SDK than `skyvault` 2.x. Upgrade each on its own version line. -> Migrating from v1? See skyvault's **[Migration Guide](docs/migrate_to_v2.md)**. V1 is in maintenance mode and will reach End of Life on October 31, 2026. +> Migrating from v1? See skyvault's **[Migration Guide](docs/migrate_to_v2.md)**. V1 is in maintenance mode and reached End of Life on October 31, 2026. +> +> Using skyvault's replaced Detect methods (`deidentifyText`, `reidentifyText`, `getDetectRun`)? See **[Migrate to the new Detect methods](docs/migrate_detect.md)**. ## Repository layout @@ -26,13 +28,14 @@ The root `package.json` (`private: true`) aggregates these npm workspaces: - `common/` — shared client, credentials, config, Detect, and error-handling code used by both `skyvault` and `flowvault` - `skyvault/` — the `skyflow-node` SDK ([README](skyvault/README.md)) - `flowvault/` — the `skyflow-flowvault-node` SDK ([README](flowvault/README.md)) -- `samples/` — runnable examples for `skyvault` ([README](samples/README.md)); `flowvault` samples live in [`flowvault/samples/`](flowvault/samples/README.md) +- `skyvault/samples/` — runnable examples for `skyvault` ([README](skyvault/samples/README.md)); `flowvault` samples live in [`flowvault/samples/`](flowvault/samples/README.md) ## Documentation - [skyvault API Reference](docs/api_reference.md) — full list of request classes, options, response classes, enums, and service-account functions - [Detect API Reference](docs/api_reference.md#detect-api) — request, response, configuration and enum types for Detect, shared by both SDKs - [Migrate from v1 to v2](docs/migrate_to_v2.md) +- [Migrate to the new Detect methods](docs/migrate_detect.md) — for skyvault users of `deidentifyText`, `reidentifyText`, `deidentifyFile` and `getDetectRun` - [Advanced initialization](docs/advanced_initialization.md) - [Authentication credentials](docs/auth_credentials.md) diff --git a/docs/api_reference.md b/docs/api_reference.md index ecb136bf..21753ac7 100644 --- a/docs/api_reference.md +++ b/docs/api_reference.md @@ -343,7 +343,7 @@ Controls how retrieved data is displayed. | Value | Description | |---|---| -| `RedactionType.DEFAULT` | Vault-configured default. | +| `RedactionType.DEFAULT` | Vault-configured default. Valid for `get` and `detokenize`; the Detect API does not accept it in a `RedactionLevel`. | | `RedactionType.PLAIN_TEXT` | Full, unmasked value. | | `RedactionType.MASKED` | Partially obscured. | | `RedactionType.REDACTED` | Fully removed. | @@ -613,7 +613,7 @@ skyvault SDK (`skyflow-node`): | Field | Type | Description | |---|---|---| | `entityName` | `EntityType` | Entity type the rule applies to. | -| `redactionType` | [`RedactionType`](#redactiontype) | `PLAIN_TEXT`, `MASKED`, `REDACTED` or `DEFAULT`. | +| `redactionType` | [`RedactionType`](#redactiontype) | `PLAIN_TEXT`, `MASKED` or `REDACTED`. The Detect API rejects `DEFAULT` here with HTTP 400. | flowvault SDK (`skyflow-flowvault-node`): diff --git a/docs/migrate_detect.md b/docs/migrate_detect.md new file mode 100644 index 00000000..9a64d68f --- /dev/null +++ b/docs/migrate_detect.md @@ -0,0 +1,399 @@ +# Migrate to the new Detect methods (skyflow-node 2.2) + +This guide is for skyvault users of the Node.js SDK (`skyflow-node`) who call the Detect methods `deidentifyText`, +`reidentifyText`, `deidentifyFile` or `getDetectRun`. Those methods, their request, options and response classes +(`DeidentifyTextRequest`, `DeidentifyTextOptions`, `TokenFormat`, `Transformations`, `DeidentifyFileOptions`, `Bleep`, +`ReidentifyTextOptions`, `GetDetectRunRequest`, `DeidentifiedTextResponse`, `ReidentifyTextResponse`) and the enums +`DetectEntities`, `TokenType` and `DetectOutputTranscription` are removed in **skyflow-node 2.2.0** and replaced by the +Detect methods documented in the [skyvault README](../skyvault/README.md#detect). Projects that call the removed +methods stop compiling on upgrade, so pin the version explicitly and follow this guide rather than relying on a +version range. + +```bash +npm install skyflow-node@2.2.0 +``` + +Client creation and the `skyflowClient.detect()` / `skyflowClient.detect(vaultId)` accessors do not change. + +> [!NOTE] +> Using a FlowVault vault? FlowVault Detect is new in `skyflow-flowvault-node` and has nothing to migrate: see +> [Detect in the flowvault README](../flowvault/README.md#detect). + +> [!WARNING] +> **`deidentifyFile` keeps its name but no longer waits for the result by default.** It submits the file and +> resolves with only the run id; `status` is `null` and nothing is written to disk until you pass `pollOptions`. +> A quick port runs without error, and the missing outputs only show up downstream. See +> [Step 4](#step-4-deidentifyfile-new-request-and-response). + +## Table of contents + +- [What changed](#what-changed) +- [Method mapping](#method-mapping) +- [Step 1: move the detection policy into a configuration](#step-1-move-the-detection-policy-into-a-configuration) +- [Step 2: deidentifyText to deidentifyString](#step-2-deidentifytext-to-deidentifystring) +- [Step 3: reidentifyText to reidentifyString](#step-3-reidentifytext-to-reidentifystring) +- [Step 4: deidentifyFile, new request and response](#step-4-deidentifyfile-new-request-and-response) +- [Step 5: getDetectRun to getRun](#step-5-getdetectrun-to-getrun) +- [New methods and options](#new-methods-and-options) +- [Migration checklist](#migration-checklist) + +## What changed + +| | Removed | Replacement | +|---|---|---| +| API | `/v1/detect/...`, one file endpoint per file type | `/v2/detect/...`, one file endpoint | +| Detection policy | an options object with setters on every call | a stored configuration id, or one inline `DetectConfiguration` object literal | +| Token type | `TokenFormat` with per-type lists | `deidentificationType` on each `Entity`; `VAULT_TOKEN` entities name their `table.column` | +| File input | `DeidentifyFileRequest(fileInput)` with a `File` or `Buffer` | `DeidentifyFileRequest({ filePath })`, `{ file }` (web `File`) or `{ dataSource, value }` | +| File polling | always, `waitTime` 1 to 64 s | opt in with `pollOptions`: `waitTime` 1 to 300 s and `maxAttempts` 1 to 23 | +| Responses | flat counts, first output only, `errors` array | `metrics`, an `output: FileOutput[]`, `location`, `DetectRunStatus` | +| New capabilities | | re-identify a file, guardrails, custom types, object entities (faces, logos), file mapping, `SKYFLOW_ID` and presigned-URL inputs | + +## Method mapping + +| Removed | Replacement | Request | Response | +|---|---|---|---| +| `deidentifyText(request, options)` | `deidentifyString(request)` | `DeidentifyStringRequest` | `DeidentifyStringResponse` | +| `reidentifyText(request, options)` | `reidentifyString(request)` | `ReidentifyStringRequest` | `ReidentifyStringResponse` | +| `deidentifyFile(request, options)` | `deidentifyFile(request)`, same name, new types | `DeidentifyFileRequest` (new shape) | `DeidentifyFileResponse` (new shape) | +| `getDetectRun(request)` | `getRun(request)` | `GetRunRequest` | `GetRunResponse` | +| — | `reidentifyFile(request)` | `ReidentifyFileRequest` | `ReidentifyFileResponse` | +| — | `checkGuardrails(request)` | `CheckGuardrailsRequest` | `CheckGuardrailsResponse` | + +The separate options argument is gone: everything is passed in the request. Every method is still reached through +`skyflowClient.detect()`, returns a promise of a response object and rejects with `SkyflowError`. + +## Step 1: move the detection policy into a configuration + +Previously every call carried the policy in an options object: `setEntities`, `setTokenFormat`, `setAllowRegexList`, +`setRestrictRegexList`, `setTransformations` and the media setters. Now the policy is a `DetectConfiguration`, +supplied in one of two ways: + +- `configurationId`: a configuration created once in Studio or through the management API, then referenced by id. + This is the recommended path. +- `configuration`: an inline `DetectConfiguration` object literal, for callers that need per-call control. + +Both are optional and sent as given. With neither, the SDK sends a configuration holding only the vault id and the run +uses the vault's default Detect settings. Setting both is rejected by the API with HTTP 400. The SDK does not create, +update or delete stored configurations. + +### Request field mapping + +| Removed (`DeidentifyTextOptions` / `DeidentifyFileOptions`) | Replacement | +|---|---| +| `setEntities(DetectEntities[])` | `detect.entities`, one `Entity` per type with `entityType` | +| `tokenFormat.setDefault(TokenType)` | An `Entity` with `entityType: EntityType.ALL` and the chosen `deidentificationType`; it applies to every type without its own rule | +| `tokenFormat.setVaultToken([...])` | An `Entity` per type with `deidentificationType: VAULT_TOKEN` and `destination: '
.'` | +| `tokenFormat.setEntityUniqueCounter([...])` | An `Entity` per type with `deidentificationType: ENTITY_UNIQUE_COUNTER` | +| `tokenFormat.setEntityOnly([...])` | An `Entity` per type with `deidentificationType: ENTITY_ONLY`; `MASK` is new | +| `setAllowRegexList` | `detect.skip` | +| `setRestrictRegexList` | `detect.restrict` | +| `transformations.setShiftDays({ max, min, entities })` | `transformation: { shiftDates: { minDays, maxDays } }` on each listed `Entity`; the entity list is gone because the rule sits on the entity | +| `setPixelDensity`, `setMaxResolution` | `media.document.pdf.density`, `media.document.pdf.maxResolution` | +| `setOutputProcessedImage`, `setOutputOcrText`, `setMaskingMethod` | `media.image` with the same three names; `MaskingMethod` values are now `BLUR`, `BLACKBOX` and `NONE` | +| `setOutputProcessedAudio`, `setOutputTranscription(DetectOutputTranscription)` | `media.audio` with the same two names; the two cannot be combined, so set `outputProcessedAudio: false` when asking for a transcription. `PLAINTEXT_TRANSCRIPTION` has no equivalent; use `TRANSCRIPTION` | +| `setBleep(Bleep)` with setters | `media.audio.bleep: { startPadding, stopPadding, frequency, gain }` | +| `new DeidentifyFileRequest(fileInput)` | `new DeidentifyFileRequest({ filePath })`, `{ file }` or `{ dataSource, value }` | +| `setWaitTime(n)` | `pollOptions: { waitTime: n, maxAttempts: m }` | +| `setOutputDirectory` | `outputDirectory`, allowed only together with `pollOptions` | +| — | New: `customTypes`, `objectEntities`, `returnEntities`, `fileMapping`, per-entity `includePrefix` / `prefixOverride`, and `SKYFLOW_ID` / `PRESIGNED_URL` inputs | + +### Enum renames + +| Removed | Replacement | +|---|---| +| `DetectEntities` (`'name'`, `'email_address'`, ..., lower-case wire values) | `EntityType` (`NAME`, `EMAIL_ADDRESS`, ..., upper-case wire values, plus `NONE`) | +| `TokenType` (`VAULT_TOKEN`, `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY`) | `DeidentificationType`, same names plus `MASK` and `UNKNOWN` | +| `DetectOutputTranscription` (five values) | `OutputTranscriptionType`: `NONE`, `TRANSCRIPTION`, `MEDICAL_TRANSCRIPTION`, `DIARIZED_TRANSCRIPTION`, `MEDICAL_DIARIZED_TRANSCRIPTION` | +| `MaskingMethod` (`Blackbox`, `Blur`) | `MaskingMethod` (`BLACKBOX`, `BLUR`, `NONE`) | +| `RedactionType` | Unchanged, still the `RedactionType` shared with `get` and `detokenize`. The Detect API does not accept `DEFAULT` in a `RedactionLevel`; use `PLAIN_TEXT`, `MASKED` or `REDACTED`. | + +### Example: the same policy before and after + +**Removed** + +```typescript +import { DeidentifyTextRequest, DeidentifyTextOptions, TokenFormat, Transformations, DetectEntities, TokenType } from 'skyflow-node'; + +const request = new DeidentifyTextRequest('My name is John Doe, and my email is johndoe@acme.com.'); +const options = new DeidentifyTextOptions(); +options.setEntities([DetectEntities.NAME, DetectEntities.EMAIL_ADDRESS, DetectEntities.DOB]); +const tokenFormat = new TokenFormat(); +tokenFormat.setDefault(TokenType.ENTITY_UNIQUE_COUNTER); +tokenFormat.setVaultToken([DetectEntities.NAME, DetectEntities.EMAIL_ADDRESS]); +options.setTokenFormat(tokenFormat); +options.setAllowRegexList(['Skyflow']); +options.setRestrictRegexList(['[0-9]{3}-[0-9]{2}-[0-9]{4}']); +const transformations = new Transformations(); +transformations.setShiftDays({ max: 30, min: 10, entities: [DetectEntities.DOB] }); +options.setTransformations(transformations); +``` + +**Replacement, inline configuration** + +```typescript +import { DeidentifyStringRequest, DetectConfiguration, EntityType, DeidentificationType } from 'skyflow-node'; + +const configuration: DetectConfiguration = { + detect: { + entities: [ + { entityType: EntityType.NAME, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: 'persons.name' }, + { entityType: EntityType.EMAIL_ADDRESS, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: 'persons.email' }, + { + entityType: EntityType.DOB, + deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER, + transformation: { shiftDates: { minDays: 10, maxDays: 30 } }, + }, + { entityType: EntityType.ALL, deidentificationType: DeidentificationType.ENTITY_UNIQUE_COUNTER }, + ], + skip: ['Skyflow'], + restrict: ['[0-9]{3}-[0-9]{2}-[0-9]{4}'], + }, +}; + +const request = new DeidentifyStringRequest('My name is John Doe, and my email is johndoe@acme.com.', { configuration }); +``` + +**Replacement, stored configuration** + +```typescript +const request = new DeidentifyStringRequest('My name is John Doe, and my email is johndoe@acme.com.', { + configurationId: '', +}); +``` + +Two points differ in meaning, not only in name: + +- The old options listed `entities` once and then assigned token types per list. Now the `deidentificationType` is + stated on each entity, and the old `setDefault` becomes an `EntityType.ALL` entry placed last. +- The old `setVaultToken` list only named the entity types. Now each `VAULT_TOKEN` entity also needs a + `destination` of the form `table.column`, the column with tokenization enabled that the value is stored in. + +## Step 2: deidentifyText to deidentifyString + +```typescript +// Removed +const before = await skyflowClient.detect().deidentifyText(request, options); +before.processedText; +for (const e of before.entities) { + e.entity; e.token; e.value; + e.textIndex?.start; e.textIndex?.end; + e.processedIndex?.start; e.processedIndex?.end; + e.scores; +} +before.wordCount; before.charCount; before.errors; + +// Replacement +const after = await skyflowClient.detect().deidentifyString(request); +after.processedText; +for (const e of after.entities) { + e.entityType; e.token; e.value; + e.location?.startIndex; e.location?.endIndex; + e.location?.startIndexProcessed; e.location?.endIndexProcessed; + e.entityScores; +} +after.metrics?.wordCount; after.metrics?.characterCount; // metrics is null when the server sends none +``` + +| Removed (`DeidentifiedTextResponse`) | Replacement (`DeidentifyStringResponse`) | +|---|---| +| `entities[].entity` | `entities[].entityType` (a string, so it can carry custom-type labels) | +| `entities[].textIndex.start` / `.end` | `entities[].location.startIndex` / `.endIndex` | +| `entities[].processedIndex.start` / `.end` | `entities[].location.startIndexProcessed` / `.endIndexProcessed` | +| `entities[].scores` | `entities[].entityScores` | +| `wordCount`, `charCount` as numbers | `metrics.wordCount`, `metrics.characterCount`, each `number | null` | +| `errors` | Removed. Failures reject the promise with `SkyflowError` | + +`value` is populated only when the configuration sets `returnEntities` to `ALL`. The default `EXCLUDE_SENSITIVE_DATA` +returns an empty value. + +## Step 3: reidentifyText to reidentifyString + +Only vault tokens can be re-identified. Placeholders from `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not +stored, so they come back unchanged. The text to re-identify is the `processedText` of a `VAULT_TOKEN` +de-identification, in which a token looks like `[NAME_daa78453-9ca9-4ba5-a3ef-91934c11196c]`. + +The old options took three lists of entity types. The new request takes a list of `RedactionLevel` rules, each naming +an entity type and a redaction type. + +```typescript +// Removed +const options = new ReidentifyTextOptions(); +options.setPlainTextEntities([DetectEntities.NAME]); +options.setMaskedEntities([DetectEntities.EMAIL_ADDRESS]); +options.setRedactedEntities([DetectEntities.SSN]); +const before = await skyflowClient.detect().reidentifyText(new ReidentifyTextRequest(''), options); + +// Replacement +const after = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest('', { // e.g. processedText of a deidentifyString call that used VAULT_TOKEN + redactionLevel: [ + { entityName: EntityType.NAME, redactionType: RedactionType.PLAIN_TEXT }, + { entityName: EntityType.EMAIL_ADDRESS, redactionType: RedactionType.MASKED }, + { entityName: EntityType.SSN, redactionType: RedactionType.REDACTED }, + ], + }), +); +``` + +| Removed list | Replacement rule | +|---|---| +| `setPlainTextEntities` | `{ entityName: x, redactionType: RedactionType.PLAIN_TEXT }` | +| `setMaskedEntities` | `{ entityName: x, redactionType: RedactionType.MASKED }` | +| `setRedactedEntities` | `{ entityName: x, redactionType: RedactionType.REDACTED }` | + +`processedText` is unchanged and `metrics` is new. Without rules every token gets its original value. The skyvault +`RedactionLevel` has only `entityName` and `redactionType`; `tokenGroupName` and `redactionPattern` belong to the +flowvault SDK. + +## Step 4: deidentifyFile, new request and response + +The method keeps its name. The request and response classes keep their names but change shape. Three things change at +once: the file input, the policy, and the polling. + +```typescript +// Removed: always polled, result in one response, processed copy written to outputDirectory +const options = new DeidentifyFileOptions(); +options.setEntities([DetectEntities.ALL]); +const tokenFormat = new TokenFormat(); +tokenFormat.setDefault(TokenType.ENTITY_UNIQUE_COUNTER); +options.setTokenFormat(tokenFormat); +options.setPixelDensity(200); +options.setMaxResolution(3000); +options.setOutputDirectory('/data/processed'); +options.setWaitTime(64); +const before = await skyflowClient.detect().deidentifyFile(new DeidentifyFileRequest(fileInput), options); +before.status; // string +before.file; // File, first output only +before.entities; // [{ file, extension }] +before.pageCount; + +// Replacement: same behaviour, opt in to polling +const after = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ + filePath: '/data/intake/patient-notes.pdf', + configurationId: '', // or configuration: { media: { document: { pdf: { density: 200, maxResolution: 3000 } } }, ... } + pollOptions: { waitTime: 64 }, + outputDirectory: '/data/processed', + }), +); +after.status; // DetectRunStatus +for (const output of after.output) { // every output, not only the first + output.processedFile; output.processedFileType; output.processedFileExtension; +} +after.metrics?.pages; + +switch (after.status) { + case DetectRunStatus.SUCCESS: // outputs are on disk under /data/processed + break; + case DetectRunStatus.IN_PROGRESS: + case DetectRunStatus.QUEUED: // the wait ran out: resume with getRun(after.runId) + break; + default: // FAILED or UNKNOWN: after.message +} +``` + +Or submit only and retrieve later, which the old method could not do: + +```typescript +const submitted = await skyflowClient.detect().deidentifyFile( + new DeidentifyFileRequest({ filePath: '/data/intake/patient-notes.pdf', configurationId: '' }), +); +const runId = submitted.runId; // status is null, nothing written to disk +``` + +### File response mapping + +| Removed (`DeidentifyFileResponse`, old shape) | Replacement (`DeidentifyFileResponse` and `GetRunResponse`) | +|---|---| +| `runId` | `runId` | +| `status` as a string | `status` as `DetectRunStatus` (`SUCCESS`, `FAILED`, `IN_PROGRESS`, `QUEUED`, `UNKNOWN`), `null` when not polled | +| `file`, `type`, `extension` for the first output | `output`, a `FileOutput[]` with `processedFile`, `processedFileType`, `processedFileExtension`; `outputType` says whether `processedFile` is base64 or a presigned URL | +| `entities` as `[{ file, extension }]` | The `FileOutput` whose `processedFileType` is `ENTITIES` (a JSON file) | +| `wordCount`, `charCount` | `metrics.wordCount`, `metrics.characterCount` | +| `sizeInKb`, `durationInSeconds` | `metrics.size`, `metrics.duration` | +| `pageCount`, `slideCount` | `metrics.pages`, `metrics.slides`; at most one of pages, slides and duration is set | +| `errors` | Removed. Failures reject with `SkyflowError`; a run that failed while processing has `status: FAILED` and `message` | +| — | `outputType`, `message` | + +### Behaviour differences to check + +- **Polling is opt in.** Without `pollOptions` the call resolves with the run id immediately. Callers that relied on the + result being complete must add `pollOptions`. +- **Budget limits changed.** `waitTime` defaults to 60 and tops out at 300, where the old method defaulted to 64 and + capped at 64. `maxAttempts` is new, default and maximum 23. +- **Elapsed time is measured by the clock.** The old loop counted only its planned delays. The new one measures real + elapsed time and never exceeds `waitTime`. +- **Output file names.** The redacted file is `processed-.` and the entities file is + `processed-entities-.json`, written under `outputDirectory` (or the working directory when none is set) + only for local file inputs that reached `SUCCESS` with base64 outputs. +- **All outputs are returned.** The old response surfaced only the first output. `output` holds the full list. +- **`UNKNOWN` ends polling.** The old method kept calling the run endpoint on any status it did not recognise. The new one + stops and returns `UNKNOWN`; check for it and call `getRun` later. +- **Validation is stricter.** `waitTime` and `maxAttempts` out of range, `outputDirectory` without `pollOptions`, and + supplying more than one of `file`, `filePath` and `value` reject before any network call. +- **Supported inputs widen.** `dataSource` can be `SKYFLOW_ID` for a file already in the vault, or `PRESIGNED_URL`, in + addition to a local file. A `Buffer` is no longer accepted directly; wrap it in a web `File` or pass base64 as `value`. + +## Step 5: getDetectRun to getRun + +```typescript +// Removed +const run = await skyflowClient.detect().getDetectRun(new GetDetectRunRequest(runId)); + +// Replacement +import * as fs from 'fs'; +const run = await skyflowClient.detect().getRun(new GetRunRequest(runId)); +if (run.status === DetectRunStatus.SUCCESS) { + for (const output of run.output) { + if (run.outputType === DeidentifySourceType.BASE64) { + const bytes = Buffer.from(output.processedFile ?? '', 'base64'); + fs.writeFileSync(`processed-${output.processedFileType}.${output.processedFileExtension?.toLowerCase()}`, bytes); + } // PRESIGNED_URL: download it yourself; do not log the URL + } +} else if (run.status === DetectRunStatus.FAILED) { + console.error('Run failed:', run.message); +} +``` + +The old call returned the file response type and wrote the processed file to disk. `getRun` returns `GetRunResponse`, +which has the same fields as the new `DeidentifyFileResponse` with `runId` echoed from the request, and writes nothing. +A blank run id is rejected before the call. + +## New methods and options + +- **`reidentifyFile(request)`**: synchronous re-identification of a file sent as base64 or a presigned URL, with the + same `RedactionLevel` rules as strings. The output comes back in the response and the SDK writes nothing to disk. + See [Re-identify a file](../skyvault/README.md#re-identify-a-file-reidentifyfilerequest). +- **`checkGuardrails(request)`**: toxicity and denied-topic screening. Toxicity is evaluated only when + `checkToxicity: true` is set; with `denyTopics` alone the response reports `toxic: false` even for toxic text, and a + request with neither is rejected with HTTP 400. See [Check guardrails](../skyvault/README.md#check-guardrails-checkguardrailsrequest). +- **`customTypes`** on `detect`: regex-defined entities with a label of your choosing, de-identified like any other + entity. +- **`objectEntities`** on `detect`: `FACE`, `LICENSE_PLATE`, `LOGO` and `SIGNATURE` detection in images. +- **`returnEntities`** on `detect`: control whether raw values come back. +- **`fileMapping`** on `DetectConfiguration`: per-column handling for structured files. +- **`processingMode`** on `pdf`: `OCR` or `TEXT_LAYER`. +- **`includePrefix` / `prefixOverride`** on `Entity` and `CustomType`: control the prefix of `VAULT_TOKEN` tokens. + `includePrefix: IncludePrefix.NO` returns raw tokens that cannot be re-identified. + +## Migration checklist + +1. Upgrade to `skyflow-node@2.2.0`. +2. Create a stored Detect configuration in Studio that reproduces your policy, or build it inline as a + `DetectConfiguration` using the field mapping above. +3. Remove the imports of the deleted classes and enums (`DeidentifyTextRequest`, `DeidentifyTextOptions`, `TokenFormat`, + `Transformations`, `DeidentifyFileOptions`, `Bleep`, `ReidentifyTextOptions`, `GetDetectRunRequest`, `DetectEntities`, + `TokenType`, `DetectOutputTranscription`). +4. Rename the calls: `deidentifyText` to `deidentifyString`, `reidentifyText` to `reidentifyString`, `getDetectRun` to + `getRun`. `deidentifyFile` keeps its name; switch its request to the object-literal form and drop the options argument. +5. Give every `VAULT_TOKEN` entity a `destination: '
.'`. +6. Add `pollOptions` to every file call that expects a completed result. +7. Replace reads of `file`, `entities` and the flat counts on file responses with a loop over `output` that branches on + `outputType`, and reads of counts with `metrics`. +8. Switch re-identify calls from the three entity lists to `redactionLevel` rules, and do not use `RedactionType.DEFAULT` + there. +9. Handle `DetectRunStatus.UNKNOWN` and a `null` status on file responses, and replace reads of `errors` with a + `try`/`catch` of `SkyflowError`. +10. Print `error.httpCode`, `message` and `error.requestId` from `SkyflowError` in your error handling. +11. Run your integration tests against the detect vault and compare outputs with the previous version for the same inputs. diff --git a/flowvault/README.md b/flowvault/README.md index a1653467..b3538525 100644 --- a/flowvault/README.md +++ b/flowvault/README.md @@ -31,7 +31,7 @@ re-identify sensitive data in text and files with Skyflow Detect. `error`, and `requestId`. Whole-call failures throw a `SkyflowError`. - **Multi-vault.** One client can hold several vault configs; pick one per call with `vault(vaultId)` or `detect(vaultId)`. -- **Detect.** The same Detect V2 API as the SkyVault SDK, scoped to a Flow DB vault — see +- **Detect.** The same Detect API as the SkyVault SDK, scoped to a Flow DB vault — see [Detect](#detect). ## Install @@ -266,7 +266,7 @@ const res = await vault.deleteFiles(req); ## Detect -Skyflow Detect finds and de-identifies sensitive data in free text and files, then re-identifies it on demand. The SDK exposes the Detect V2 API through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault, and the controller honours the same credentials, [timeouts and retries](#timeouts-and-retries) as `vault()`. +Skyflow Detect finds and de-identifies sensitive data in strings and files (documents, spreadsheets, images and audio), re-identifies vault tokens on demand, and screens text against guardrails for toxicity and denied topics. The SDK exposes it through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault, and the controller honours the same credentials, [timeouts and retries](#timeouts-and-retries) as `vault()`. Detect configurations are created in Studio or through the management API. The SDK does not create, update or delete them. @@ -284,7 +284,12 @@ Detect configurations are created in Studio or through the management API. The S - **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy that grants the Detokenization action on the destination, unless it is a Vault Owner. - **Configuration.** `configurationId` is the id of a Detect configuration created through the management API. The alternative is an inline `DetectConfiguration` passed as `configuration` (see [Detect configuration](#detect-configuration)). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings. Sending both is rejected by the API with HTTP 400. - **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the token group named by `destination`. That token group must already exist in the vault; Detect writes into it but does not create it. -- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](../docs/api_reference.md#detect-api). +- **Where the values come from.** + - ``: open the configuration in Studio under your vault's Detect settings and copy its id, or take the `id` returned when you create one through the management API. + - ``: a token group defined in the vault schema (Studio: vault > **Schema**). Each token group names the tokenization settings that `VAULT_TOKEN` values are stored with. + - ``: a redaction pattern configured on the vault (Studio: vault > **Settings**), referenced by name in a `RedactionLevel`. + - If you do not have a vault yet, start at [Try Skyflow](https://skyflow.com/try-skyflow); the [Skyflow docs](https://docs.skyflow.com) cover Studio, token groups and redaction patterns. +- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](../docs/api_reference.md#detect-api). Errors are described under [Detect errors](#detect-errors). ### De-identify a string: `.deidentifyString(request)` @@ -336,7 +341,32 @@ for (const entity of inlineResponse.entities) { console.log(inlineResponse.metrics?.wordCount); ``` -`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. +`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. As JSON: + +```json +{ + "processedText": "My name is [NAME_1], and my email is [EMAIL_ADDRESS_1].", + "entities": [ + { + "token": "[NAME_1]", + "value": "", + "location": { "startIndex": 11, "endIndex": 19, "startIndexProcessed": 11, "endIndexProcessed": 19 }, + "entityType": "NAME", + "entityScores": { "NAME": 0.99 } + }, + { + "token": "[EMAIL_ADDRESS_1]", + "value": "", + "location": { "startIndex": 37, "endIndex": 53, "startIndexProcessed": 37, "endIndexProcessed": 54 }, + "entityType": "EMAIL_ADDRESS", + "entityScores": { "EMAIL_ADDRESS": 0.99 } + } + ], + "metrics": { "wordCount": 11, "characterCount": 55, "size": null, "pages": null, "slides": null, "duration": null } +} +``` + +`value` is populated only when the configuration sets `returnEntities` to `ALL`. > [!TIP] > See the full example in the samples directory: [deidentify-string.ts](samples/detect-api/deidentify-string.ts) @@ -399,6 +429,9 @@ const configuration: DetectConfiguration = { File de-identification is asynchronous on the API: the submit call returns a run id and the result is retrieved from the run. `deidentifyFile` exposes both modes. +> [!WARNING] +> **`deidentifyFile` does not wait for the result unless you set `pollOptions`.** Without it the call resolves with only the run id: `status` is `null`, `output` is empty and nothing is written to disk. Code that reads the outputs straight after the call runs without error, and the missing files only show up downstream. Set `pollOptions` to wait for the run, or keep the run id and call [`getRun`](#get-run-getrunrequest). + - **Submit only** (default): one call, returns `runId`. `status` is `null` and nothing is written to disk. Retrieve the result later with [`getRun`](#get-run-getrunrequest). - **Submit and poll**: set `pollOptions`. The SDK polls the run with exponential backoff (1, 2, 4, 8, 16, then 32 seconds between lookups) until it is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `waitTime` (default 60 s, max 300) or `maxAttempts` (default 23, max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED` so you can resume with `getRun`. @@ -450,22 +483,85 @@ Outputs are written only when polling reached `SUCCESS`, the request was built f > [!TIP] > See the full examples in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](samples/detect-api/deidentify-file-async.ts) +The submit-only response and a polled response that reached `SUCCESS`, as JSON: + +```json +{ "runId": "48f7613c-b4c4-478a-a42f-dde537f54b6c", "status": null, "outputType": null, "output": [], "metrics": null, "message": null } +``` + +```json +{ + "runId": "48f7613c-b4c4-478a-a42f-dde537f54b6c", + "status": "SUCCESS", + "outputType": "BASE64", + "output": [ + { "processedFile": "JVBERi0xLjcK...", "processedFileType": "REDACTED_FILE", "processedFileExtension": "PDF" }, + { "processedFile": "eyJlbnRpdGllcyI6W3...", "processedFileType": "ENTITIES", "processedFileExtension": "JSON" } + ], + "metrics": { "pages": 3, "wordCount": 842, "characterCount": 5120, "size": 118.4, "slides": null, "duration": null }, + "message": null +} +``` + +For audio, `outputTranscription` cannot be combined with `outputProcessedAudio`: the API rejects the pair with HTTP 400. Set `outputProcessedAudio: false` in the audio configuration when you want a transcription. + ### Re-identify a string: `.reidentifyString(request)` Re-identification looks tokens up in the vault, so it applies to values that were de-identified with `VAULT_TOKEN`. Placeholders produced by `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not stored anywhere and cannot be re-identified. `ReidentifyStringRequest` takes the tokenized text and an optional list of `RedactionLevel` rules. For flowvault vaults each rule targets a `tokenGroupName` and applies a named `redactionPattern` configured in the vault. Without rules every token is replaced by its original value. +The text to re-identify is the `processedText` of a de-identification that used `VAULT_TOKEN`. A vault token looks like `[NAME_QUo9AjT]`, not like the `[NAME_1]` counter placeholder, so the round trip looks like this: + ```typescript -import { ReidentifyStringRequest, ReidentifyStringResponse } from 'skyflow-flowvault-node'; +import { + DeidentifyStringRequest, + ReidentifyStringRequest, + ReidentifyStringResponse, + EntityType, + DeidentificationType, +} from 'skyflow-flowvault-node'; -const response: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( - new ReidentifyStringRequest('My name is [NAME_1].', { +// 1. De-identify with VAULT_TOKEN: each value is stored in its token group and replaced by its vault token. +const deidentified = await skyflowClient.detect().deidentifyString( + new DeidentifyStringRequest('My name is John Doe and my email is johndoe@acme.com.', { + configuration: { + detect: { + entities: [ + { entityType: EntityType.NAME, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: '' }, + { entityType: EntityType.EMAIL_ADDRESS, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: '' }, + ], + }, + }, + }), +); +const tokenized = deidentified.processedText ?? ''; +// My name is [NAME_QUo9AjT] and my email is [EMAIL_ADDRESS_x7Fp2Kd]. + +// 2. Re-identify: every vault token is replaced by its original value. +const original: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest(tokenized), +); +console.log(original.processedText); +// My name is John Doe and my email is johndoe@acme.com. + +// 3. Or apply a redaction pattern to one token group. +const rendered = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest(tokenized, { redactionLevel: [{ tokenGroupName: '', redactionPattern: '' }], }), ); -console.log(response.processedText); -// My name is J**n D*e. +console.log(rendered.processedText); +// My name is J**n D*e and my email is johndoe@acme.com. +``` + +`ReidentifyStringResponse` carries `processedText` and `metrics`. As JSON: + +```json +{ + "processedText": "My name is J**n D*e and my email is johndoe@acme.com.", + "metrics": { "wordCount": 10, "characterCount": 53, "size": null, "pages": null, "slides": null, "duration": null } +} ``` `RedactionLevel` is the flowvault SDK's own type and has only `tokenGroupName` and `redactionPattern`; the `entityName` and `redactionType` rules of the skyvault SDK do not exist here. Both fields are optional on the API, so the SDK passes the list through as given and the server rejects invalid combinations. @@ -489,13 +585,28 @@ const response = await skyflowClient.detect().reidentifyFile( if (response.status === DetectRunStatus.SUCCESS) { for (const output of response.output) { - // output.processedFile is base64 or a presigned URL, per response.outputType - const processed = Buffer.from(output.processedFile ?? '', 'base64'); + if (response.outputType === ReidentifySourceType.BASE64) { + const processed = Buffer.from(output.processedFile ?? '', 'base64'); + fs.writeFileSync(`reidentified.${output.processedFileExtension?.toLowerCase()}`, processed); + } else { + // PRESIGNED_URL: output.processedFile is a URL to download; do not log it. + } } } ``` -`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. +`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. As JSON: + +```json +{ + "status": "SUCCESS", + "outputType": "BASE64", + "output": [ + { "processedFile": "TXkgbmFtZSBpcyBKb2huIERvZS4=", "processedFileType": "REIDENTIFIED_FILE", "processedFileExtension": "TXT" } + ], + "metrics": { "wordCount": 5, "characterCount": 20, "size": 0.02, "pages": null, "slides": null, "duration": null } +} +``` > [!TIP] > See the full example in the samples directory: [reidentify-file.ts](samples/detect-api/reidentify-file.ts) @@ -505,16 +616,22 @@ if (response.status === DetectRunStatus.SUCCESS) { File de-identification is asynchronous: the submit call returns a run id, and `getRun` retrieves the run's status and, once it has succeeded, its outputs. `output` is empty and `outputType` and `metrics` are `null` until the run completes. ```typescript -import { GetRunRequest, GetRunResponse, DetectRunStatus } from 'skyflow-flowvault-node'; +import * as fs from 'fs'; +import { GetRunRequest, GetRunResponse, DetectRunStatus, DeidentifySourceType } from 'skyflow-flowvault-node'; const run: GetRunResponse = await skyflowClient.detect().getRun(new GetRunRequest('')); switch (run.status) { case DetectRunStatus.SUCCESS: for (const output of run.output) { - // output.processedFile is base64 or a presigned URL, per run.outputType // output.processedFileType is e.g. REDACTED_FILE or ENTITIES // output.processedFileExtension is a FileDataFormat such as PDF + if (run.outputType === DeidentifySourceType.BASE64) { + const bytes = Buffer.from(output.processedFile ?? '', 'base64'); + fs.writeFileSync(`processed-${output.processedFileType}.${output.processedFileExtension?.toLowerCase()}`, bytes); + } else { + // PRESIGNED_URL: output.processedFile is a URL to download; do not log it. + } } break; case DetectRunStatus.FAILED: @@ -525,12 +642,14 @@ switch (run.status) { } ``` +`GetRunResponse` has the same fields as `DeidentifyFileResponse` (`runId`, `status`, `outputType`, `output`, `metrics`, `message`); see the JSON under [De-identify a file](#de-identify-a-file-deidentifyfilerequest). `getRun` never writes to disk. + > [!TIP] > See the full example in the samples directory: [get-run.ts](samples/detect-api/get-run.ts) ### Check guardrails: `.checkGuardrails(request)` -`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. +`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The server evaluates toxicity only when `checkToxicity: true` is sent; with `denyTopics` alone, `toxic` is `false` even for toxic text, so set `checkToxicity: true` explicitly whenever you want that check. A request with neither field is rejected with HTTP 400. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. ```typescript import { CheckGuardrailsRequest, GuardrailsValidation } from 'skyflow-flowvault-node'; @@ -544,11 +663,45 @@ if (response.validation === GuardrailsValidation.FAILED) { } ``` -All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling). Client-side validation fails before any network call. +`CheckGuardrailsResponse` as JSON: + +```json +{ "text": "", "toxic": true, "deniedTopic": false, "validation": "FAILED" } +``` > [!TIP] > See the full example in the samples directory: [check-guardrails.ts](samples/detect-api/check-guardrails.ts) +### Detect errors + +All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling); `error.httpCode`, `message` and `error.requestId` identify the failure, and the request id is what Skyflow support asks for. Client-side validation fails before any network call with HTTP code 400 and no request id. Server-side failures carry the API's message, for example a `checkGuardrails` call with neither `checkToxicity` nor `denyTopics`: + +```typescript +import { CheckGuardrailsRequest, SkyflowError } from 'skyflow-flowvault-node'; + +try { + await skyflowClient.detect().checkGuardrails(new CheckGuardrailsRequest(userPrompt)); +} catch (err) { + if (err instanceof SkyflowError) { + console.error(`Error during checkGuardrails: [${err.error?.httpCode}] ${err.message} (requestId=${err.error?.requestId})`); + } +} +``` + +```json +{ + "error": { + "grpc_code": 3, + "http_code": 400, + "message": "Guardrails are not provided. Please provide 'check_toxicity' or 'deny_topics' in the request body.", + "http_status": "Bad Request", + "details": [] + } +} +``` + +A `deidentifyFile` run that was accepted but failed while processing does not throw: `status` is `FAILED` and `message` carries the detail. + ## Custom request headers The SDK supports a fixed set of custom headers, listed on the `CustomHeaderKey` diff --git a/flowvault/samples/detect-api/reidentify-string.ts b/flowvault/samples/detect-api/reidentify-string.ts index 34233d43..edc8ae69 100644 --- a/flowvault/samples/detect-api/reidentify-string.ts +++ b/flowvault/samples/detect-api/reidentify-string.ts @@ -6,19 +6,26 @@ import { SkyflowConfig, VaultConfig, SkyflowError, + DeidentifyStringRequest, + DeidentifyStringResponse, ReidentifyStringRequest, ReidentifyStringResponse, + EntityType, + DeidentificationType, } from 'skyflow-flowvault-node'; /** - * Skyflow Detect: re-identify a string + * Skyflow Detect: the de-identify / re-identify round trip * - * Re-identification looks tokens up in the vault, so it applies to values that were - * de-identified with VAULT_TOKEN. This example demonstrates how to: + * Re-identification looks tokens up in the vault, so it applies only to values that were + * de-identified with VAULT_TOKEN. Such a token looks like [NAME_QUo9AjT]; counter placeholders + * such as [NAME_1] are never stored and cannot be re-identified. + * This example demonstrates how to: * 1. Configure credentials and a vault - * 2. Re-identify tokenized text, restoring the original values - * 3. Apply redaction rules so some entities come back masked or redacted - * 4. Handle errors + * 2. De-identify text with VAULT_TOKEN so the values are stored in a token group + * 3. Re-identify the tokenized text, restoring the original values + * 4. Apply a redaction pattern to a token group + * 5. Handle errors */ async function performReidentifyString() { try { @@ -42,29 +49,58 @@ async function performReidentifyString() { }; const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - const tokenizedText = 'My name is [NAME_1], and my SSN is [SSN_1].'; + // Step 4: De-identify with VAULT_TOKEN. Each value is stored in the token group named by + // destination and replaced by its vault token. + const deidentifyRequest = new DeidentifyStringRequest( + 'My name is John Doe and my email is johndoe@acme.com.', + { + configuration: { + detect: { + entities: [ + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '', // Replace with a token group of the vault + }, + { + entityType: EntityType.EMAIL_ADDRESS, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '', // Replace with a token group of the vault + }, + ], + }, + }, + }, + ); + const deidentified: DeidentifyStringResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyString(deidentifyRequest); + const tokenizedText = deidentified.processedText ?? ''; + console.log('De-identified text:', tokenizedText); + // My name is [NAME_QUo9AjT] and my email is [EMAIL_ADDRESS_x7Fp2Kd]. - // Step 4: Without rules every token is replaced by its original value + // Step 5: Without rules every vault token is replaced by its original value const plainRequest = new ReidentifyStringRequest(tokenizedText); const plainResponse: ReidentifyStringResponse = await skyflowClient .detect(primaryVaultConfig.vaultId) .reidentifyString(plainRequest); console.log('Re-identified text:', plainResponse.processedText); - // Step 5: Redaction rules. Each rule targets a token group and applies a redaction pattern configured in the vault. + // Step 6: Redaction rules. Each rule targets a token group and applies a redaction pattern + // configured in the vault. const redactedRequest = new ReidentifyStringRequest(tokenizedText, { redactionLevel: [ { tokenGroupName: '', redactionPattern: '' }, - { tokenGroupName: '', redactionPattern: '' }, ], }); const redactedResponse = await skyflowClient .detect(primaryVaultConfig.vaultId) .reidentifyString(redactedRequest); console.log('Re-identified text with redaction rules:', redactedResponse.processedText); + // My name is J**n D*e and my email is johndoe@acme.com. console.log('Character count:', redactedResponse.metrics?.characterCount); } catch (error) { - // Comprehensive error handling + // Comprehensive error handling. Quote the request id when contacting Skyflow support. if (error instanceof SkyflowError) { console.error('Skyflow Specific Error:', { httpCode: error.error?.httpCode, @@ -80,5 +116,5 @@ async function performReidentifyString() { } } -// Invoke the re-identify string function +// Invoke the round trip performReidentifyString(); diff --git a/samples/detect-api/reidentify-string.ts b/samples/detect-api/reidentify-string.ts deleted file mode 100644 index 1b7e3cbd..00000000 --- a/samples/detect-api/reidentify-string.ts +++ /dev/null @@ -1,86 +0,0 @@ -import { - Credentials, - Env, - LogLevel, - Skyflow, - SkyflowConfig, - VaultConfig, - SkyflowError, - ReidentifyStringRequest, - ReidentifyStringResponse, - EntityType, - RedactionType, -} from 'skyflow-node'; - -/** - * Skyflow Detect: re-identify a string - * - * Re-identification looks tokens up in the vault, so it applies to values that were - * de-identified with VAULT_TOKEN. This example demonstrates how to: - * 1. Configure credentials and a vault - * 2. Re-identify tokenized text, restoring the original values - * 3. Apply redaction rules so some entities come back masked or redacted - * 4. Handle errors - */ -async function performReidentifyString() { - try { - // Step 1: Configure credentials - const credentials: Credentials = { - path: 'path-to-credentials-json', // Path to credentials file - }; - - // Step 2: Configure the vault - const primaryVaultConfig: VaultConfig = { - vaultId: '', // Unique vault identifier - clusterId: '', // From the vault URL - env: Env.PROD, // Deployment environment - credentials: credentials, // Authentication method - }; - - // Step 3: Configure and initialize the Skyflow client - const skyflowConfig: SkyflowConfig = { - vaultConfigs: [primaryVaultConfig], - logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production - }; - const skyflowClient: Skyflow = new Skyflow(skyflowConfig); - - const tokenizedText = 'My name is [NAME_1], and my SSN is [SSN_1].'; - - // Step 4: Without rules every token is replaced by its original value - const plainRequest = new ReidentifyStringRequest(tokenizedText); - const plainResponse: ReidentifyStringResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .reidentifyString(plainRequest); - console.log('Re-identified text:', plainResponse.processedText); - - // Step 5: Redaction rules. Each rule names an entity and how to render it. - const redactedRequest = new ReidentifyStringRequest(tokenizedText, { - redactionLevel: [ - { entityName: EntityType.NAME, redactionType: RedactionType.MASKED }, - { entityName: EntityType.SSN, redactionType: RedactionType.REDACTED }, - ], - }); - const redactedResponse = await skyflowClient - .detect(primaryVaultConfig.vaultId) - .reidentifyString(redactedRequest); - console.log('Re-identified text with redaction rules:', redactedResponse.processedText); - console.log('Character count:', redactedResponse.metrics?.characterCount); - } catch (error) { - // Comprehensive error handling - if (error instanceof SkyflowError) { - console.error('Skyflow Specific Error:', { - httpCode: error.error?.httpCode, - grpcCode: error.error?.grpcCode, - httpStatus: error.error?.httpStatus, - message: error.message, - details: error.error?.details, - requestId: error.error?.requestId, - }); - } else { - console.error('Unexpected Error:', error); - } - } -} - -// Invoke the re-identify string function -performReidentifyString(); diff --git a/skyvault/README.md b/skyvault/README.md index 5bc7ba73..e9fe2381 100644 --- a/skyvault/README.md +++ b/skyvault/README.md @@ -4,7 +4,9 @@ > > Meant for **Privacy DB** vaults. Working with a Flow DB vault? Use the [FlowVault SDK](../flowvault/README.md) (`skyflow-flowvault-node`) instead. > -> Migrating from v1? See the **[Migration Guide](../docs/migrate_to_v2.md)** for step-by-step instructions. V1 is in maintenance mode and will reach End of Life on October 31, 2026. +> Migrating from v1? See the **[Migration Guide](../docs/migrate_to_v2.md)** for step-by-step instructions. V1 is in maintenance mode and reached End of Life on October 31, 2026. +> +> Using the Detect methods that this SDK replaced (`deidentifyText`, `reidentifyText`, `getDetectRun`)? See **[Migrate to the new Detect methods](../docs/migrate_detect.md)**. Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow SDK for Node.js, Deno, Bun, and Cloudflare Workers. @@ -56,6 +58,7 @@ Securely handle sensitive data at rest, in-transit, and in-use with the Skyflow - [Re-identify a file: `.reidentifyFile(request)`](#re-identify-a-file-reidentifyfilerequest) - [Get run: `.getRun(request)`](#get-run-getrunrequest) - [Check guardrails: `.checkGuardrails(request)`](#check-guardrails-checkguardrailsrequest) + - [Detect errors](#detect-errors) - [Connections](#connections) - [Configure a connection](#configure-a-connection) - [Invoke a connection](#invoke-a-connection) @@ -92,7 +95,8 @@ The Skyflow SDK enables you to connect to your Skyflow Vault(s) to securely hand > [!IMPORTANT] > This readme documents SDK version 2. > For version 1 see the [v1.14.2 README](https://github.com/skyflowapi/skyflow-node/tree/1.14.2). -> For more information on how to migrate see [MIGRATE_TO_V2.md](../docs/migrate_to_v2.md). +> For more information on how to migrate see [MIGRATE_TO_V2.md](../docs/migrate_to_v2.md). +> For the Detect methods replaced in 2.2.0 see [migrate_detect.md](../docs/migrate_detect.md). ## Installation @@ -238,6 +242,8 @@ Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f Upgrade from `skyflow-node` v1 using the dedicated guide in [docs/migrate_to_v2.md](../docs/migrate_to_v2.md). +Already on v2 and using `deidentifyText`, `reidentifyText`, `deidentifyFile` or `getDetectRun`? Those are replaced by the [Detect](#detect) methods in 2.2.0; see **[Migrate to the new Detect methods](../docs/migrate_detect.md)**. + ## Vault The [Vault](https://docs.skyflow.com/docs/vaults) performs operations on the vault such as inserting records, detokenizing tokens, retrieving tokens for list of `skyflowId`s and to invoke the Connection. @@ -280,7 +286,7 @@ Insert response: { insertedFields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f Set the `continueOnError` flag to `true` to allow insert operations to proceed despite encountering partial errors. > [!TIP] -> See the full example in the samples directory: [insert-continue-on-error.ts](../samples/vault-api/insert-continue-on-error.ts) +> See the full example in the samples directory: [insert-continue-on-error.ts](samples/vault-api/insert-continue-on-error.ts) #### Upsert request @@ -334,7 +340,7 @@ Detokenization response: { detokenizedFields: [{ token: '5391-4629-3722-7102', v ``` > [!TIP] -> See the full example in the samples directory: [detokenzie-records.ts](../samples/vault-api/detokenzie-records.ts) +> See the full example in the samples directory: [detokenzie-records.ts](samples/vault-api/detokenzie-records.ts) ### Get Record(s): `.get(request, options)` @@ -416,7 +422,7 @@ getOptions.setReturnTokens(true); // Set to `true` to get tokens ``` > [!TIP] -> See the full example in the samples directory: [get-records.ts](../samples/vault-api/get-records.ts) +> See the full example in the samples directory: [get-records.ts](samples/vault-api/get-records.ts) #### Get by column name and column values @@ -431,7 +437,7 @@ const getRequest: GetColumnRequest = new GetColumnRequest( ``` > [!TIP] -> See the full example in the samples directory: [get-column-values.ts](../samples/vault-api/get-column-values.ts) +> See the full example in the samples directory: [get-column-values.ts](samples/vault-api/get-column-values.ts) #### Redaction Types @@ -490,7 +496,7 @@ Update response: { updatedField: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5 For all `UpdateOptions` setters and `TokenMode` values, see [UpdateOptions](../docs/api_reference.md#updateoptions) in the API reference. > [!TIP] -> See the full example in the samples directory: [update-record.ts](../samples/vault-api/update-record.ts) +> See the full example in the samples directory: [update-record.ts](samples/vault-api/update-record.ts) ### Delete Records @@ -519,7 +525,7 @@ Delete response: { deletedIds: ['a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c'], errors: ``` > [!TIP] -> See the full example in the samples directory: [delete-records.ts](../samples/vault-api/delete-records.ts) +> See the full example in the samples directory: [delete-records.ts](samples/vault-api/delete-records.ts) ### Query @@ -548,7 +554,7 @@ Query response: { fields: [{ skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', ``` > [!TIP] -> See the full example in the samples directory: [query-records.ts](../samples/vault-api/query-records.ts) +> See the full example in the samples directory: [query-records.ts](samples/vault-api/query-records.ts) Refer to [Query your data](https://docs.skyflow.com/query-data/) and [Execute Query](https://docs.skyflow.com/record/#QueryService_ExecuteQuery) for guidelines and restrictions on supported SQL statements, operators, and keywords. @@ -598,7 +604,7 @@ File upload: { skyflowId: 'a8f0c2e1-7b3d-4f9a-8c21-1d2e3f4a5b6c', errors: null } For all `FileUploadOptions` setters, see [FileUploadOptions](../docs/api_reference.md#fileuploadoptions) in the API reference. > [!TIP] -> See the full example in the samples directory: [file-upload.ts](../samples/vault-api/file-upload.ts) +> See the full example in the samples directory: [file-upload.ts](samples/vault-api/file-upload.ts) ### Retrieve Existing Tokens: `.tokenize(request)` @@ -628,13 +634,13 @@ Tokenization Result: { tokens: ['5391-4629-3722-7102', '0f6b8a2c-90ab-4cde-9def- ``` > [!TIP] -> See the full example in the samples directory: [tokenize-records.ts](../samples/vault-api/tokenize-records.ts) +> See the full example in the samples directory: [tokenize-records.ts](samples/vault-api/tokenize-records.ts) ## Detect -Skyflow Detect finds and de-identifies sensitive data in free text and files, then re-identifies it on demand. The SDK exposes the Detect V2 API through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault and uses the vault's credentials and log level. +Skyflow Detect finds and de-identifies sensitive data in strings and files (documents, spreadsheets, images and audio), re-identifies vault tokens on demand, and screens text against guardrails for toxicity and denied topics. The SDK exposes it through `DetectController`, returned by `skyflowClient.detect()` and `skyflowClient.detect(vaultId)`. Every request is scoped to that vault and uses the vault's credentials and log level. -Detect configurations are created in Studio or through the management API. The SDK does not create, update or delete them. +Detect configurations are created in Studio or through the management API. The SDK does not create, update or delete them. These methods replace `deidentifyText`, `reidentifyText`, `deidentifyFile` (old shape) and `getDetectRun`; see [Migrate to the new Detect methods](../docs/migrate_detect.md) if you call those. | Method | Request | Response | Description | |--------|---------|----------|-------------| @@ -650,7 +656,12 @@ Detect configurations are created in Studio or through the management API. The S - **Role.** The service account needs the Detect Invoker, Detect Manager or Vault Owner role on the vault. Re-identifying values stored as `VAULT_TOKEN` is governed like detokenization: the caller also needs a policy that grants the Detokenization action on the destination table and columns, unless it is a Vault Owner. - **Configuration.** `configurationId` is the id of a Detect configuration created through the management API. The alternative is an inline `DetectConfiguration` passed as `configuration` (see [Detect configuration](#detect-configuration)). Both are optional: with neither set the SDK sends only the vault id and the run uses the vault's default Detect settings. Sending both is rejected by the API with HTTP 400. - **Destination for vault tokens.** An entity de-identified with `VAULT_TOKEN` stores its value in the `table.column` named by `destination`. That table and column must already exist in the vault. -- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](../docs/api_reference.md#detect-api). +- **Where the values come from.** + - ``: open the configuration in Studio under your vault's Detect settings and copy its id, or take the `id` returned when you create one through the management API. + - `
.`: a table and column of the vault schema (Studio: vault > **Schema**) with tokenization enabled; `VAULT_TOKEN` values are stored there and re-identified from there. + - `RedactionType`: `PLAIN_TEXT`, `MASKED` or `REDACTED`, the same values `get` and `detokenize` use. The Detect API does not accept `DEFAULT` here. + - If you do not have a vault yet, start at [Try Skyflow](https://skyflow.com/try-skyflow); the [Skyflow docs](https://docs.skyflow.com) cover Studio, schemas and tokenization. +- **API reference.** Every Detect request, response and configuration type is listed in the [API reference](../docs/api_reference.md#detect-api). Errors are described under [Detect errors](#detect-errors). ### De-identify a string: `.deidentifyString(request)` @@ -702,10 +713,35 @@ for (const entity of inlineResponse.entities) { console.log(inlineResponse.metrics?.wordCount); ``` -`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. +`DeidentifyStringResponse` carries `processedText`, `entities` (a `DetectedEntity[]` with token, entity type, location and confidence scores; never `null`) and `metrics`. As JSON: + +```json +{ + "processedText": "My name is [NAME_1], and my email is [EMAIL_ADDRESS_1].", + "entities": [ + { + "token": "[NAME_1]", + "value": "", + "location": { "startIndex": 11, "endIndex": 19, "startIndexProcessed": 11, "endIndexProcessed": 19 }, + "entityType": "NAME", + "entityScores": { "NAME": 0.99 } + }, + { + "token": "[EMAIL_ADDRESS_1]", + "value": "", + "location": { "startIndex": 37, "endIndex": 53, "startIndexProcessed": 37, "endIndexProcessed": 54 }, + "entityType": "EMAIL_ADDRESS", + "entityScores": { "EMAIL_ADDRESS": 0.99 } + } + ], + "metrics": { "wordCount": 11, "characterCount": 55, "size": null, "pages": null, "slides": null, "duration": null } +} +``` + +`value` is populated only when the configuration sets `returnEntities` to `ALL`. > [!TIP] -> See the full example in the samples directory: [deidentify-string.ts](../samples/detect-api/deidentify-string.ts) +> See the full example in the samples directory: [deidentify-string.ts](samples/detect-api/deidentify-string.ts) ### Detect configuration @@ -765,6 +801,9 @@ const configuration: DetectConfiguration = { File de-identification is asynchronous on the API: the submit call returns a run id and the result is retrieved from the run. `deidentifyFile` exposes both modes. +> [!WARNING] +> **`deidentifyFile` does not wait for the result unless you set `pollOptions`.** Without it the call resolves with only the run id: `status` is `null`, `output` is empty and nothing is written to disk. Code that reads the outputs straight after the call runs without error, and the missing files only show up downstream. Set `pollOptions` to wait for the run, or keep the run id and call [`getRun`](#get-run-getrunrequest). If you are migrating from `deidentifyText`-era code that always waited, see [Migrate to the new Detect methods](../docs/migrate_detect.md#step-4-deidentifyfile-new-request-and-response). + - **Submit only** (default): one call, returns `runId`. `status` is `null` and nothing is written to disk. Retrieve the result later with [`getRun`](#get-run-getrunrequest). - **Submit and poll**: set `pollOptions`. The SDK polls the run with exponential backoff (1, 2, 4, 8, 16, then 32 seconds between lookups) until it is `SUCCESS`, `FAILED` or `UNKNOWN`, or until `waitTime` (default 60 s, max 300) or `maxAttempts` (default 23, max 23) is used up. If the budget runs out the response carries the run id with `IN_PROGRESS` or `QUEUED` so you can resume with `getRun`. @@ -813,31 +852,98 @@ switch (response.status) { Outputs are written only when polling reached `SUCCESS`, the request was built from a local file, and the run delivered them as base64. A request built from a `SKYFLOW_ID` or `PRESIGNED_URL` has no local name to derive from, and presigned-URL outputs are not fetched, so in those cases read `output` and write the files yourself. Files are named `processed-.`, with the entities output as `processed-entities-.json`; `outputDirectory` is created if missing and defaults to the working directory. Writing outputs needs the Node.js file system and is not supported in Cloudflare Workers. +The submit-only response and a polled response that reached `SUCCESS`, as JSON: + +```json +{ "runId": "48f7613c-b4c4-478a-a42f-dde537f54b6c", "status": null, "outputType": null, "output": [], "metrics": null, "message": null } +``` + +```json +{ + "runId": "48f7613c-b4c4-478a-a42f-dde537f54b6c", + "status": "SUCCESS", + "outputType": "BASE64", + "output": [ + { "processedFile": "JVBERi0xLjcK...", "processedFileType": "REDACTED_FILE", "processedFileExtension": "PDF" }, + { "processedFile": "eyJlbnRpdGllcyI6W3...", "processedFileType": "ENTITIES", "processedFileExtension": "JSON" } + ], + "metrics": { "pages": 3, "wordCount": 842, "characterCount": 5120, "size": 118.4, "slides": null, "duration": null }, + "message": null +} +``` + +For audio, `outputTranscription` cannot be combined with `outputProcessedAudio`: the API rejects the pair with HTTP 400. Set `outputProcessedAudio: false` in the audio configuration when you want a transcription. + > [!TIP] -> See the full examples in the samples directory: [deidentify-file.ts](../samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](../samples/detect-api/deidentify-file-async.ts) +> See the full examples in the samples directory: [deidentify-file.ts](samples/detect-api/deidentify-file.ts) and [deidentify-file-async.ts](samples/detect-api/deidentify-file-async.ts) ### Re-identify a string: `.reidentifyString(request)` Re-identification looks tokens up in the vault, so it applies to values that were de-identified with `VAULT_TOKEN`. Placeholders produced by `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY` or `MASK` are not stored anywhere and cannot be re-identified. -`ReidentifyStringRequest` takes the tokenized text and an optional list of `RedactionLevel` rules. For skyvault vaults each rule names an `entityName` and the `redactionType` to render it with. Without rules every token is replaced by its original value. +`ReidentifyStringRequest` takes the tokenized text and an optional list of `RedactionLevel` rules. For skyvault vaults each rule names an `entityName` and the `redactionType` to render it with: `PLAIN_TEXT` reveals the original value, `MASKED` masks it and `REDACTED` keeps it hidden. Without rules every token is replaced by its original value. + +The text to re-identify is the `processedText` of a de-identification that used `VAULT_TOKEN`. A vault token carries the token of the value stored in the `table.column` destination, for example `[NAME_daa78453-9ca9-4ba5-a3ef-91934c11196c]`, not the `[NAME_1]` counter placeholder, so the round trip looks like this: ```typescript -import { ReidentifyStringRequest, ReidentifyStringResponse, EntityType, RedactionType } from 'skyflow-node'; +import { + DeidentifyStringRequest, + ReidentifyStringRequest, + ReidentifyStringResponse, + EntityType, + DeidentificationType, + RedactionType, +} from 'skyflow-node'; -const response: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( - new ReidentifyStringRequest('My name is [NAME_1].', { - redactionLevel: [{ entityName: EntityType.NAME, redactionType: RedactionType.MASKED }], +// 1. De-identify with VAULT_TOKEN: each value is stored in
. and replaced by its vault token. +const deidentified = await skyflowClient.detect().deidentifyString( + new DeidentifyStringRequest('My name is John Doe and my email is johndoe@acme.com.', { + configuration: { + detect: { + entities: [ + { entityType: EntityType.NAME, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: '
.' }, + { entityType: EntityType.EMAIL_ADDRESS, deidentificationType: DeidentificationType.VAULT_TOKEN, destination: '
.' }, + ], + }, + }, }), ); -console.log(response.processedText); -// My name is J**n D*e. +const tokenized = deidentified.processedText ?? ''; +// My name is [NAME_daa78453-9ca9-4ba5-a3ef-91934c11196c] and my email is [EMAIL_ADDRESS_3f0c…]. + +// 2. Re-identify: every vault token is replaced by its original value. +const original: ReidentifyStringResponse = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest(tokenized), +); +console.log(original.processedText); +// My name is John Doe and my email is johndoe@acme.com. + +// 3. Or choose how each entity type is rendered. +const rendered = await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest(tokenized, { + redactionLevel: [ + { entityName: EntityType.NAME, redactionType: RedactionType.PLAIN_TEXT }, + { entityName: EntityType.EMAIL_ADDRESS, redactionType: RedactionType.MASKED }, + ], + }), +); +console.log(rendered.processedText); +// My name is John Doe and my email is j*******@acme.com. +``` + +`ReidentifyStringResponse` carries `processedText` and `metrics`. As JSON: + +```json +{ + "processedText": "My name is John Doe and my email is j*******@acme.com.", + "metrics": { "wordCount": 10, "characterCount": 54, "size": null, "pages": null, "slides": null, "duration": null } +} ``` `RedactionLevel` is the skyvault SDK's own type and has only `entityName` and `redactionType`; the `tokenGroupName` and `redactionPattern` rules of the flowvault SDK do not exist here. Both fields are optional on the API, so the SDK passes the list through as given and the server rejects invalid combinations. > [!TIP] -> See the full example in the samples directory: [reidentify-string.ts](../samples/detect-api/reidentify-string.ts) +> See the full example in the samples directory: [reidentify-string.ts](samples/detect-api/reidentify-string.ts) ### Re-identify a file: `.reidentifyFile(request)` @@ -855,32 +961,53 @@ const response = await skyflowClient.detect().reidentifyFile( if (response.status === DetectRunStatus.SUCCESS) { for (const output of response.output) { - // output.processedFile is base64 or a presigned URL, per response.outputType - const processed = Buffer.from(output.processedFile ?? '', 'base64'); + if (response.outputType === ReidentifySourceType.BASE64) { + const processed = Buffer.from(output.processedFile ?? '', 'base64'); + fs.writeFileSync(`reidentified.${output.processedFileExtension?.toLowerCase()}`, processed); + } else { + // PRESIGNED_URL: output.processedFile is a URL to download; do not log it. + } } } ``` -`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. +`ReidentifyFileResponse` carries `status`, `outputType`, `output` (never `null`) and `metrics`. As JSON: + +```json +{ + "status": "SUCCESS", + "outputType": "BASE64", + "output": [ + { "processedFile": "TXkgbmFtZSBpcyBKb2huIERvZS4=", "processedFileType": "REIDENTIFIED_FILE", "processedFileExtension": "TXT" } + ], + "metrics": { "wordCount": 5, "characterCount": 20, "size": 0.02, "pages": null, "slides": null, "duration": null } +} +``` > [!TIP] -> See the full example in the samples directory: [reidentify-file.ts](../samples/detect-api/reidentify-file.ts) +> See the full example in the samples directory: [reidentify-file.ts](samples/detect-api/reidentify-file.ts) ### Get run: `.getRun(request)` File de-identification is asynchronous: the submit call returns a run id, and `getRun` retrieves the run's status and, once it has succeeded, its outputs. `output` is empty and `outputType` and `metrics` are `null` until the run completes. ```typescript -import { GetRunRequest, GetRunResponse, DetectRunStatus } from 'skyflow-node'; +import * as fs from 'fs'; +import { GetRunRequest, GetRunResponse, DetectRunStatus, DeidentifySourceType } from 'skyflow-node'; const run: GetRunResponse = await skyflowClient.detect().getRun(new GetRunRequest('')); switch (run.status) { case DetectRunStatus.SUCCESS: for (const output of run.output) { - // output.processedFile is base64 or a presigned URL, per run.outputType // output.processedFileType is e.g. REDACTED_FILE or ENTITIES // output.processedFileExtension is a FileDataFormat such as PDF + if (run.outputType === DeidentifySourceType.BASE64) { + const bytes = Buffer.from(output.processedFile ?? '', 'base64'); + fs.writeFileSync(`processed-${output.processedFileType}.${output.processedFileExtension?.toLowerCase()}`, bytes); + } else { + // PRESIGNED_URL: output.processedFile is a URL to download; do not log it. + } } break; case DetectRunStatus.FAILED: @@ -891,12 +1018,14 @@ switch (run.status) { } ``` +`GetRunResponse` has the same fields as `DeidentifyFileResponse` (`runId`, `status`, `outputType`, `output`, `metrics`, `message`); see the JSON under [De-identify a file](#de-identify-a-file-deidentifyfilerequest). `getRun` never writes to disk. + > [!TIP] -> See the full example in the samples directory: [get-run.ts](../samples/detect-api/get-run.ts) +> See the full example in the samples directory: [get-run.ts](samples/detect-api/get-run.ts) ### Check guardrails: `.checkGuardrails(request)` -`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. +`checkGuardrails` screens text for toxicity and for a list of denied topics, which makes it a natural pre-LLM check. `checkToxicity` and `denyTopics` are optional: the SDK sends each one only when you set it. The server evaluates toxicity only when `checkToxicity: true` is sent; with `denyTopics` alone, `toxic` is `false` even for toxic text, so set `checkToxicity: true` explicitly whenever you want that check. A request with neither field is rejected with HTTP 400. The response is passed through as returned, so `toxic`, `deniedTopic` and `validation` carry exactly what the API sent back. ```typescript import { CheckGuardrailsRequest, GuardrailsValidation } from 'skyflow-node'; @@ -910,10 +1039,48 @@ if (response.validation === GuardrailsValidation.FAILED) { } ``` -All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling). Client-side validation fails before any network call. +`CheckGuardrailsResponse` as JSON: + +```json +{ "text": "", "toxic": true, "deniedTopic": false, "validation": "FAILED" } +``` > [!TIP] -> See the full example in the samples directory: [check-guardrails.ts](../samples/detect-api/check-guardrails.ts) +> See the full example in the samples directory: [check-guardrails.ts](samples/detect-api/check-guardrails.ts) + +### Detect errors + +All Detect methods throw `SkyflowError` as described in [Error handling](#error-handling); `error.httpCode`, `message` and `error.requestId` identify the failure, and the request id is what Skyflow support asks for. Client-side validation fails before any network call with HTTP code 400 and no request id. Server-side failures carry the API's message, for example a `RedactionLevel` that uses `RedactionType.DEFAULT`: + +```typescript +import { ReidentifyStringRequest, EntityType, RedactionType, SkyflowError } from 'skyflow-node'; + +try { + await skyflowClient.detect().reidentifyString( + new ReidentifyStringRequest(tokenized, { + redactionLevel: [{ entityName: EntityType.NAME, redactionType: RedactionType.DEFAULT }], + }), + ); +} catch (err) { + if (err instanceof SkyflowError) { + console.error(`Error during reidentifyString: [${err.error?.httpCode}] ${err.message} (requestId=${err.error?.requestId})`); + } +} +``` + +```json +{ + "error": { + "grpc_code": 3, + "http_code": 400, + "message": "ReidentifyStringRequestV2 failed. redaction_type in redaction_level[0] must be REDACTED, MASKED, or PLAIN_TEXT", + "http_status": "Bad Request", + "details": [] + } +} +``` + +A `deidentifyFile` run that was accepted but failed while processing does not throw: `status` is `FAILED` and `message` carries the detail. ## Connections @@ -980,7 +1147,7 @@ Invoke connection response: { data: { status: 'success', transaction_id: 'txn_ab **pathParams, queryParams, header, body** are the JSON objects represented as dictionaries that will be sent through the connection integration url. > [!TIP] -> See the full example in the samples directory: [scoped-token-generation-example.ts](../samples/vault-api/invoke-connection.ts) +> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/vault-api/invoke-connection.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on integrations with Connections, Functions, and Pipelines. ## Authentication & authorization @@ -1112,7 +1279,7 @@ const options: BearerTokenOptions = { ``` > [!TIP] -> See the full example in the samples directory: [scoped-token-generation-example.ts](../samples/service-account/scoped-token-generation-example.ts) +> See the full example in the samples directory: [scoped-token-generation-example.ts](samples/service-account/scoped-token-generation-example.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. #### Generate bearer tokens with `ctx` for context-aware authorization @@ -1167,7 +1334,7 @@ const credentials: PathCredentials = { ``` > [!TIP] -> See the full example in the samples directory: [token-generation-with-context-example.ts](../samples/service-account/token-generation-with-context-example.ts) +> See the full example in the samples directory: [token-generation-with-context-example.ts](samples/service-account/token-generation-with-context-example.ts) > See Skyflow's [context-aware authorization](https://docs.skyflow.com) and [conditional data access](https://docs.skyflow.com) docs for policy variable syntax like `request.context.*`. #### Generate signed data tokens @@ -1224,7 +1391,7 @@ type SignedDataTokensOptions = { ``` > [!TIP] -> See the full example in the samples directory: [signed-token-generation-example.ts](../samples/service-account/signed-token-generation-example.ts) +> See the full example in the samples directory: [signed-token-generation-example.ts](samples/service-account/signed-token-generation-example.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. ## Client management @@ -1349,7 +1516,7 @@ message: Authentication failed. Bearer token is expired. Use a valid bearer toke If you encounter this kind of error, retry the request. During the retry the SDK detects that the previous bearer token has expired and generates a new one for the current and subsequent requests. > [!TIP] -> See the full example in the samples directory: [bearer-token-expiry-example.ts](../samples/service-account/bearer-token-expiry-example.ts) +> See the full example in the samples directory: [bearer-token-expiry-example.ts](samples/service-account/bearer-token-expiry-example.ts) > See [docs.skyflow.com](https://docs.skyflow.com) for more details on authentication, access control, and governance for Skyflow. ## Troubleshooting diff --git a/samples/README.md b/skyvault/samples/README.md similarity index 99% rename from samples/README.md rename to skyvault/samples/README.md index 6c7adf36..7fc4c47f 100644 --- a/samples/README.md +++ b/skyvault/samples/README.md @@ -57,7 +57,7 @@ Sample files are organized by API type: 3. Run the sample: ```bash - cd samples + cd skyvault/samples npm install ts-node vault-api/insert-records.ts ``` diff --git a/samples/deprecated/service-account/bearer-token-expiry-example.ts b/skyvault/samples/deprecated/service-account/bearer-token-expiry-example.ts similarity index 100% rename from samples/deprecated/service-account/bearer-token-expiry-example.ts rename to skyvault/samples/deprecated/service-account/bearer-token-expiry-example.ts diff --git a/samples/deprecated/service-account/scoped-token-generation-example.ts b/skyvault/samples/deprecated/service-account/scoped-token-generation-example.ts similarity index 100% rename from samples/deprecated/service-account/scoped-token-generation-example.ts rename to skyvault/samples/deprecated/service-account/scoped-token-generation-example.ts diff --git a/samples/deprecated/vault-api/credentials-options.ts b/skyvault/samples/deprecated/vault-api/credentials-options.ts similarity index 100% rename from samples/deprecated/vault-api/credentials-options.ts rename to skyvault/samples/deprecated/vault-api/credentials-options.ts diff --git a/samples/deprecated/vault-api/detokenize-records.ts b/skyvault/samples/deprecated/vault-api/detokenize-records.ts similarity index 100% rename from samples/deprecated/vault-api/detokenize-records.ts rename to skyvault/samples/deprecated/vault-api/detokenize-records.ts diff --git a/samples/deprecated/vault-api/file-upload.ts b/skyvault/samples/deprecated/vault-api/file-upload.ts similarity index 100% rename from samples/deprecated/vault-api/file-upload.ts rename to skyvault/samples/deprecated/vault-api/file-upload.ts diff --git a/samples/deprecated/vault-api/get-records.ts b/skyvault/samples/deprecated/vault-api/get-records.ts similarity index 100% rename from samples/deprecated/vault-api/get-records.ts rename to skyvault/samples/deprecated/vault-api/get-records.ts diff --git a/samples/deprecated/vault-api/update-record.ts b/skyvault/samples/deprecated/vault-api/update-record.ts similarity index 100% rename from samples/deprecated/vault-api/update-record.ts rename to skyvault/samples/deprecated/vault-api/update-record.ts diff --git a/samples/detect-api/check-guardrails.ts b/skyvault/samples/detect-api/check-guardrails.ts similarity index 100% rename from samples/detect-api/check-guardrails.ts rename to skyvault/samples/detect-api/check-guardrails.ts diff --git a/samples/detect-api/deidentify-file-async.ts b/skyvault/samples/detect-api/deidentify-file-async.ts similarity index 100% rename from samples/detect-api/deidentify-file-async.ts rename to skyvault/samples/detect-api/deidentify-file-async.ts diff --git a/samples/detect-api/deidentify-file.ts b/skyvault/samples/detect-api/deidentify-file.ts similarity index 100% rename from samples/detect-api/deidentify-file.ts rename to skyvault/samples/detect-api/deidentify-file.ts diff --git a/samples/detect-api/deidentify-string.ts b/skyvault/samples/detect-api/deidentify-string.ts similarity index 100% rename from samples/detect-api/deidentify-string.ts rename to skyvault/samples/detect-api/deidentify-string.ts diff --git a/samples/detect-api/get-run.ts b/skyvault/samples/detect-api/get-run.ts similarity index 100% rename from samples/detect-api/get-run.ts rename to skyvault/samples/detect-api/get-run.ts diff --git a/samples/detect-api/reidentify-file.ts b/skyvault/samples/detect-api/reidentify-file.ts similarity index 100% rename from samples/detect-api/reidentify-file.ts rename to skyvault/samples/detect-api/reidentify-file.ts diff --git a/skyvault/samples/detect-api/reidentify-string.ts b/skyvault/samples/detect-api/reidentify-string.ts new file mode 100644 index 00000000..643a0c85 --- /dev/null +++ b/skyvault/samples/detect-api/reidentify-string.ts @@ -0,0 +1,122 @@ +import { + Credentials, + Env, + LogLevel, + Skyflow, + SkyflowConfig, + VaultConfig, + SkyflowError, + DeidentifyStringRequest, + DeidentifyStringResponse, + ReidentifyStringRequest, + ReidentifyStringResponse, + EntityType, + DeidentificationType, + RedactionType, +} from 'skyflow-node'; + +/** + * Skyflow Detect: the de-identify / re-identify round trip + * + * Re-identification looks tokens up in the vault, so it applies only to values that were + * de-identified with VAULT_TOKEN. Such a token looks like [NAME_daa78453-9ca9-4ba5-a3ef-91934c11196c]; + * counter placeholders such as [NAME_1] are never stored and cannot be re-identified. + * This example demonstrates how to: + * 1. Configure credentials and a vault + * 2. De-identify text with VAULT_TOKEN so the values are stored in a table.column + * 3. Re-identify the tokenized text, restoring the original values + * 4. Apply redaction rules so some entities come back masked + * 5. Handle errors + */ +async function performReidentifyString() { + try { + // Step 1: Configure credentials + const credentials: Credentials = { + path: 'path-to-credentials-json', // Path to credentials file + }; + + // Step 2: Configure the vault + const primaryVaultConfig: VaultConfig = { + vaultId: '', // Unique vault identifier + clusterId: '', // From the vault URL + env: Env.PROD, // Deployment environment + credentials: credentials, // Authentication method + }; + + // Step 3: Configure and initialize the Skyflow client + const skyflowConfig: SkyflowConfig = { + vaultConfigs: [primaryVaultConfig], + logLevel: LogLevel.INFO, // Recommended to use LogLevel.ERROR in production + }; + const skyflowClient: Skyflow = new Skyflow(skyflowConfig); + + // Step 4: De-identify with VAULT_TOKEN. Each value is stored in
. (a column with + // tokenization enabled) and replaced by its vault token. + const deidentifyRequest = new DeidentifyStringRequest( + 'My name is John Doe and my email is johndoe@acme.com.', + { + configuration: { + detect: { + entities: [ + { + entityType: EntityType.NAME, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '
.', // Replace with table.column + }, + { + entityType: EntityType.EMAIL_ADDRESS, + deidentificationType: DeidentificationType.VAULT_TOKEN, + destination: '
.', // Replace with table.column + }, + ], + }, + }, + }, + ); + const deidentified: DeidentifyStringResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .deidentifyString(deidentifyRequest); + const tokenizedText = deidentified.processedText ?? ''; + console.log('De-identified text:', tokenizedText); + // My name is [NAME_daa78453-9ca9-4ba5-a3ef-91934c11196c] and my email is [EMAIL_ADDRESS_3f0c...]. + + // Step 5: Without rules every vault token is replaced by its original value + const plainRequest = new ReidentifyStringRequest(tokenizedText); + const plainResponse: ReidentifyStringResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyString(plainRequest); + console.log('Re-identified text:', plainResponse.processedText); + + // Step 6: Redaction rules. Each rule names an entity type and how to render it + // (PLAIN_TEXT, MASKED or REDACTED). + const redactedRequest = new ReidentifyStringRequest(tokenizedText, { + redactionLevel: [ + { entityName: EntityType.NAME, redactionType: RedactionType.PLAIN_TEXT }, + { entityName: EntityType.EMAIL_ADDRESS, redactionType: RedactionType.MASKED }, + ], + }); + const redactedResponse = await skyflowClient + .detect(primaryVaultConfig.vaultId) + .reidentifyString(redactedRequest); + console.log('Re-identified text with redaction rules:', redactedResponse.processedText); + // My name is John Doe and my email is j*******@acme.com. + console.log('Character count:', redactedResponse.metrics?.characterCount); + } catch (error) { + // Comprehensive error handling. Quote the request id when contacting Skyflow support. + if (error instanceof SkyflowError) { + console.error('Skyflow Specific Error:', { + httpCode: error.error?.httpCode, + grpcCode: error.error?.grpcCode, + httpStatus: error.error?.httpStatus, + message: error.message, + details: error.error?.details, + requestId: error.error?.requestId, + }); + } else { + console.error('Unexpected Error:', error); + } + } +} + +// Invoke the round trip +performReidentifyString(); diff --git a/samples/package-lock.json b/skyvault/samples/package-lock.json similarity index 100% rename from samples/package-lock.json rename to skyvault/samples/package-lock.json diff --git a/samples/package.json b/skyvault/samples/package.json similarity index 100% rename from samples/package.json rename to skyvault/samples/package.json diff --git a/samples/service-account/bearer-token-expiry-example.ts b/skyvault/samples/service-account/bearer-token-expiry-example.ts similarity index 100% rename from samples/service-account/bearer-token-expiry-example.ts rename to skyvault/samples/service-account/bearer-token-expiry-example.ts diff --git a/samples/service-account/scoped-token-generation-example.ts b/skyvault/samples/service-account/scoped-token-generation-example.ts similarity index 100% rename from samples/service-account/scoped-token-generation-example.ts rename to skyvault/samples/service-account/scoped-token-generation-example.ts diff --git a/samples/service-account/signed-token-generation-example.ts b/skyvault/samples/service-account/signed-token-generation-example.ts similarity index 100% rename from samples/service-account/signed-token-generation-example.ts rename to skyvault/samples/service-account/signed-token-generation-example.ts diff --git a/samples/service-account/token-generation-example.ts b/skyvault/samples/service-account/token-generation-example.ts similarity index 100% rename from samples/service-account/token-generation-example.ts rename to skyvault/samples/service-account/token-generation-example.ts diff --git a/samples/service-account/token-generation-with-context-example.ts b/skyvault/samples/service-account/token-generation-with-context-example.ts similarity index 100% rename from samples/service-account/token-generation-with-context-example.ts rename to skyvault/samples/service-account/token-generation-with-context-example.ts diff --git a/samples/vault-api/.env b/skyvault/samples/vault-api/.env similarity index 100% rename from samples/vault-api/.env rename to skyvault/samples/vault-api/.env diff --git a/samples/vault-api/client-operations.ts b/skyvault/samples/vault-api/client-operations.ts similarity index 100% rename from samples/vault-api/client-operations.ts rename to skyvault/samples/vault-api/client-operations.ts diff --git a/samples/vault-api/credentials-options.ts b/skyvault/samples/vault-api/credentials-options.ts similarity index 100% rename from samples/vault-api/credentials-options.ts rename to skyvault/samples/vault-api/credentials-options.ts diff --git a/samples/vault-api/data-residency.ts b/skyvault/samples/vault-api/data-residency.ts similarity index 100% rename from samples/vault-api/data-residency.ts rename to skyvault/samples/vault-api/data-residency.ts diff --git a/samples/vault-api/delete-records.ts b/skyvault/samples/vault-api/delete-records.ts similarity index 100% rename from samples/vault-api/delete-records.ts rename to skyvault/samples/vault-api/delete-records.ts diff --git a/samples/vault-api/detokenzie-records.ts b/skyvault/samples/vault-api/detokenzie-records.ts similarity index 100% rename from samples/vault-api/detokenzie-records.ts rename to skyvault/samples/vault-api/detokenzie-records.ts diff --git a/samples/vault-api/file-upload.ts b/skyvault/samples/vault-api/file-upload.ts similarity index 100% rename from samples/vault-api/file-upload.ts rename to skyvault/samples/vault-api/file-upload.ts diff --git a/samples/vault-api/get-column-values.ts b/skyvault/samples/vault-api/get-column-values.ts similarity index 100% rename from samples/vault-api/get-column-values.ts rename to skyvault/samples/vault-api/get-column-values.ts diff --git a/samples/vault-api/get-records.ts b/skyvault/samples/vault-api/get-records.ts similarity index 100% rename from samples/vault-api/get-records.ts rename to skyvault/samples/vault-api/get-records.ts diff --git a/samples/vault-api/insert-byot.ts b/skyvault/samples/vault-api/insert-byot.ts similarity index 100% rename from samples/vault-api/insert-byot.ts rename to skyvault/samples/vault-api/insert-byot.ts diff --git a/samples/vault-api/insert-continue-on-error.ts b/skyvault/samples/vault-api/insert-continue-on-error.ts similarity index 100% rename from samples/vault-api/insert-continue-on-error.ts rename to skyvault/samples/vault-api/insert-continue-on-error.ts diff --git a/samples/vault-api/insert-records.ts b/skyvault/samples/vault-api/insert-records.ts similarity index 100% rename from samples/vault-api/insert-records.ts rename to skyvault/samples/vault-api/insert-records.ts diff --git a/samples/vault-api/invoke-connection.ts b/skyvault/samples/vault-api/invoke-connection.ts similarity index 100% rename from samples/vault-api/invoke-connection.ts rename to skyvault/samples/vault-api/invoke-connection.ts diff --git a/samples/vault-api/query-records.ts b/skyvault/samples/vault-api/query-records.ts similarity index 100% rename from samples/vault-api/query-records.ts rename to skyvault/samples/vault-api/query-records.ts diff --git a/samples/vault-api/tokenize-records.ts b/skyvault/samples/vault-api/tokenize-records.ts similarity index 100% rename from samples/vault-api/tokenize-records.ts rename to skyvault/samples/vault-api/tokenize-records.ts diff --git a/samples/vault-api/update-record.ts b/skyvault/samples/vault-api/update-record.ts similarity index 100% rename from samples/vault-api/update-record.ts rename to skyvault/samples/vault-api/update-record.ts From 89bfb823015cc3bcc40b17aa7cc46aa9af809585 Mon Sep 17 00:00:00 2001 From: skyflow-bharti Date: Tue, 10 Nov 2026 17:29:50 +0530 Subject: [PATCH 10/11] SK-3076 update data format name --- common/src/detect/internal/request-mapper.ts | 8 +- common/src/detect/internal/response-mapper.ts | 27 +- .../model/request/deidentify-file/index.ts | 10 +- .../model/request/reidentify-file/index.ts | 10 +- .../model/response/reidentify-file/index.ts | 6 +- common/src/detect/types.ts | 38 +- common/test/detect/edge-cases.test.js | 6 +- common/test/detect/file-format-enums.test.js | 45 + common/test/detect/file-output-writer.test.js | 16 +- common/test/detect/models.test.js | 11 +- common/test/detect/request-mapper.test.js | 23 +- common/test/detect/response-mapper.test.js | 6 +- docs/api_reference.md | 11 +- docs/migrate_detect.md | 11 +- flowvault/README.md | 8 +- .../api-report/skyflow-flowvault-node.api.md | 2327 +++++++++++++++-- .../samples/detect-api/deidentify-file.ts | 4 +- flowvault/samples/detect-api/get-run.ts | 2 +- .../samples/detect-api/reidentify-file.ts | 4 +- skyvault/README.md | 8 +- skyvault/api-report/skyflow-node.api.md | 1160 ++++++-- .../samples/detect-api/deidentify-file.ts | 4 +- skyvault/samples/detect-api/get-run.ts | 2 +- .../samples/detect-api/reidentify-file.ts | 4 +- skyvault/src/index.ts | 8 +- 25 files changed, 3289 insertions(+), 470 deletions(-) create mode 100644 common/test/detect/file-format-enums.test.js diff --git a/common/src/detect/internal/request-mapper.ts b/common/src/detect/internal/request-mapper.ts index e0911894..4e228b26 100644 --- a/common/src/detect/internal/request-mapper.ts +++ b/common/src/detect/internal/request-mapper.ts @@ -38,7 +38,7 @@ import { DetectConfiguration, Document, Entity, - FileDataFormat, + DeidentifyFileDataFormat, FileMapping, Image, IncludePrefix, @@ -108,13 +108,13 @@ async function encodeLocalFile(request: DeidentifyFileRequest, logLevel: LogLeve } /** The format matching the file extension, or undefined when unknown so the server infers it. @internal */ -export function inferFormat(fileName: string | null): FileDataFormat | undefined { +export function inferFormat(fileName: string | null): DeidentifyFileDataFormat | undefined { if (!fileName) { return undefined; } const ext = path.extname(fileName).replace(/^\./, "").toUpperCase(); - return ext && Object.prototype.hasOwnProperty.call(FileDataFormat, ext) - ? (FileDataFormat as Record)[ext] + return ext && Object.prototype.hasOwnProperty.call(DeidentifyFileDataFormat, ext) + ? (DeidentifyFileDataFormat as Record)[ext] : undefined; } diff --git a/common/src/detect/internal/response-mapper.ts b/common/src/detect/internal/response-mapper.ts index c21e91da..58b9f7b8 100644 --- a/common/src/detect/internal/response-mapper.ts +++ b/common/src/detect/internal/response-mapper.ts @@ -13,12 +13,14 @@ import { import { DeidentifyFileSource, DetectRunStatus, + DeidentifyFileDataFormat, DetectedEntity, EntityLocation, - FileDataFormat, FileOutput, GuardrailsValidation, Metrics, + ReidentifyFileDataFormat, + ReidentifyFileOutput, ReidentifyFileSource, } from "../types"; import DeidentifyStringResponse from "../model/response/deidentify-string"; @@ -112,7 +114,7 @@ export function toFileOutput(output: FileOutputV2): FileOutput { processedFile: orNull(output.processedFile), processedFileType: output.processedFileType === undefined || output.processedFileType === null ? null : String(output.processedFileType), processedFileExtension: enumOrDefault( - FileDataFormat, + DeidentifyFileDataFormat, output.processedFileExtension === undefined || output.processedFileExtension === null ? undefined : String(output.processedFileExtension).toUpperCase(), @@ -125,6 +127,25 @@ function toFileOutputs(outputs: FileOutputV2[] | undefined | null): FileOutput[] return (outputs ?? []).filter((o) => o != null).map(toFileOutput); } +/** The re-identify counterpart of `toFileOutput`: the extension is a `ReidentifyFileDataFormat`. @internal */ +export function toReidentifyFileOutput(output: FileOutputV2): ReidentifyFileOutput { + return { + processedFile: orNull(output.processedFile), + processedFileType: output.processedFileType === undefined || output.processedFileType === null ? null : String(output.processedFileType), + processedFileExtension: enumOrDefault( + ReidentifyFileDataFormat, + output.processedFileExtension === undefined || output.processedFileExtension === null + ? undefined + : String(output.processedFileExtension).toUpperCase(), + null, + ), + }; +} + +function toReidentifyFileOutputs(outputs: FileOutputV2[] | undefined | null): ReidentifyFileOutput[] { + return (outputs ?? []).filter((o) => o != null).map(toReidentifyFileOutput); +} + /** @internal */ export function toGetRunResponse( runId: string, @@ -166,7 +187,7 @@ export function toReidentifyFileResponse( return new ReidentifyFileResponse({ status: enumOrDefault(DetectRunStatus, response.status, DetectRunStatus.UNKNOWN) as DetectRunStatus, outputType: resolveSource(response.outputType === undefined || response.outputType === null ? null : String(response.outputType)), - output: toFileOutputs(response.output), + output: toReidentifyFileOutputs(response.output), metrics: toMetrics(response.metrics), }); } diff --git a/common/src/detect/model/request/deidentify-file/index.ts b/common/src/detect/model/request/deidentify-file/index.ts index 036a7a90..6beed043 100644 --- a/common/src/detect/model/request/deidentify-file/index.ts +++ b/common/src/detect/model/request/deidentify-file/index.ts @@ -1,4 +1,4 @@ -import { DeidentifyFileSource, DetectConfiguration, FileDataFormat, PollOptions } from "../../../types"; +import { DeidentifyFileSource, DetectConfiguration, DeidentifyFileDataFormat, PollOptions } from "../../../types"; /** * Fields of a {@link DeidentifyFileRequest}. Supply the file through exactly @@ -14,7 +14,7 @@ export interface DeidentifyFileRequestFields { /** Path to a local file; read and base64-encoded when sent. */ filePath?: string; /** File format. Inferred from the file name's extension when omitted for a local file. */ - dataFormat?: FileDataFormat; + dataFormat?: DeidentifyFileDataFormat; /** * Id of a Detect configuration created through the management API. * Sending it together with `configuration` is rejected by the API. @@ -44,7 +44,7 @@ class DeidentifyFileRequest { private _filePath?: string; - private _dataFormat?: FileDataFormat; + private _dataFormat?: DeidentifyFileDataFormat; private _configurationId?: string; @@ -98,11 +98,11 @@ class DeidentifyFileRequest { this._filePath = value; } - get dataFormat(): FileDataFormat | undefined { + get dataFormat(): DeidentifyFileDataFormat | undefined { return this._dataFormat; } - set dataFormat(value: FileDataFormat | undefined) { + set dataFormat(value: DeidentifyFileDataFormat | undefined) { this._dataFormat = value; } diff --git a/common/src/detect/model/request/reidentify-file/index.ts b/common/src/detect/model/request/reidentify-file/index.ts index adbd2320..0cc2be5f 100644 --- a/common/src/detect/model/request/reidentify-file/index.ts +++ b/common/src/detect/model/request/reidentify-file/index.ts @@ -1,4 +1,4 @@ -import { FileDataFormat, RedactionRule, ReidentifyFileSource } from "../../../types"; +import { ReidentifyFileDataFormat, RedactionRule, ReidentifyFileSource } from "../../../types"; /** Optional fields of a {@link ReidentifyFileRequest}. */ export interface ReidentifyFileRequestFields { @@ -6,7 +6,7 @@ export interface ReidentifyFileRequestFields { * File format. Limited to the formats that support `VAULT_TOKEN`: `TXT`, * `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` or `PPTX`. */ - dataFormat?: FileDataFormat; + dataFormat?: ReidentifyFileDataFormat; /** Redaction rules, as your SDK's `RedactionLevel`. Without rules every token is replaced by its original value. */ redactionLevel?: RedactionRule[]; } @@ -17,7 +17,7 @@ class ReidentifyFileRequest { private _value: string; - private _dataFormat?: FileDataFormat; + private _dataFormat?: ReidentifyFileDataFormat; private _redactionLevel?: RedactionRule[]; @@ -46,11 +46,11 @@ class ReidentifyFileRequest { this._value = value; } - get dataFormat(): FileDataFormat | undefined { + get dataFormat(): ReidentifyFileDataFormat | undefined { return this._dataFormat; } - set dataFormat(value: FileDataFormat | undefined) { + set dataFormat(value: ReidentifyFileDataFormat | undefined) { this._dataFormat = value; } diff --git a/common/src/detect/model/response/reidentify-file/index.ts b/common/src/detect/model/response/reidentify-file/index.ts index 7d15ed54..f1f53948 100644 --- a/common/src/detect/model/response/reidentify-file/index.ts +++ b/common/src/detect/model/response/reidentify-file/index.ts @@ -1,4 +1,4 @@ -import { DetectRunStatus, FileOutput, Metrics, ReidentifyFileSource } from "../../../types"; +import { DetectRunStatus, ReidentifyFileOutput, Metrics, ReidentifyFileSource } from "../../../types"; /** Response of `detect().reidentifyFile()`. */ class ReidentifyFileResponse { @@ -9,7 +9,7 @@ class ReidentifyFileResponse { readonly outputType: ReidentifyFileSource | null; /** Re-identified file outputs, never null. */ - readonly output: FileOutput[]; + readonly output: ReidentifyFileOutput[]; /** Size and counts. */ readonly metrics: Metrics | null; @@ -17,7 +17,7 @@ class ReidentifyFileResponse { constructor(input: { status: DetectRunStatus; outputType: ReidentifyFileSource | null; - output: FileOutput[]; + output: ReidentifyFileOutput[]; metrics: Metrics | null; }) { this.status = input.status; diff --git a/common/src/detect/types.ts b/common/src/detect/types.ts index fbc94925..635c6373 100644 --- a/common/src/detect/types.ts +++ b/common/src/detect/types.ts @@ -122,8 +122,12 @@ export type ReidentifyFileSource = string; /** The wire value every SDK uses for a base64 input or output. @internal */ export const BASE64_SOURCE = "BASE64"; -/** File formats Skyflow Detect accepts and produces. */ -export enum FileDataFormat { +/** + * File formats accepted by `deidentifyFile` as the request `dataFormat`, and the format reported on every + * `FileOutput` of `deidentifyFile` and `getRun`. Mirrors the API's `DeidentifyFileRequestV2.dataFormat` and + * `FileOutput.processedFileExtension` enums. The wire value is the lower-case extension. + */ +export enum DeidentifyFileDataFormat { BMP = "BMP", CSV = "CSV", DCM = "DCM", @@ -149,6 +153,22 @@ export enum FileDataFormat { ZIP = "ZIP", } +/** + * File formats accepted by `reidentifyFile` as the request `dataFormat`, and the format reported on every + * `ReidentifyFileOutput` of its response: the formats that support the `VAULT_TOKEN` de-identification type. + * Mirrors the API's `ReidentifyFileRequestV2.dataFormat` enum, which is narrower than `DeidentifyFileDataFormat`. + */ +export enum ReidentifyFileDataFormat { + CSV = "CSV", + DOCX = "DOCX", + JSON = "JSON", + JSONL = "JSONL", + PPTX = "PPTX", + TXT = "TXT", + XLSX = "XLSX", + XML = "XML", +} + /** Status of a Detect file run. `SUCCESS`, `FAILED` and `UNKNOWN` are terminal. */ export enum DetectRunStatus { UNKNOWN = "UNKNOWN", @@ -433,11 +453,21 @@ export interface Metrics { duration: number | null; } -/** One output of a file run. */ +/** One output of a de-identification file run, on `DeidentifyFileResponse` and `GetRunResponse`. */ export interface FileOutput { /** Base64 content or a presigned URL, per the response's `outputType`. */ processedFile: string | null; /** `REDACTED_FILE`, `ENTITIES`, `OBJECT_ENTITIES`, `REDACTED_TEXT`, and so on. */ processedFileType: string | null; - processedFileExtension: FileDataFormat | null; + processedFileExtension: DeidentifyFileDataFormat | null; +} + +/** One output of `reidentifyFile`: the same shape as `FileOutput`, with a `ReidentifyFileDataFormat` extension. */ +export interface ReidentifyFileOutput { + /** Base64 content or a presigned URL, per the response's `outputType`. */ + processedFile: string | null; + /** `REIDENTIFIED_FILE`, and so on. */ + processedFileType: string | null; + /** Null when not reported or not a re-identification format. */ + processedFileExtension: ReidentifyFileDataFormat | null; } diff --git a/common/test/detect/edge-cases.test.js b/common/test/detect/edge-cases.test.js index 468d6f75..aca0eb4a 100644 --- a/common/test/detect/edge-cases.test.js +++ b/common/test/detect/edge-cases.test.js @@ -29,7 +29,7 @@ import { pollRun } from '../../src/detect/internal/run-poller'; import { writeOutputs } from '../../src/detect/internal/file-output-writer'; import { toDetectConfig } from '../../src/detect/internal/request-mapper'; import { completeDeidentifyFile } from '../../src/detect/internal/deidentify-file-flow'; -import { DetectRunStatus, FileDataFormat, ObjectEntityType } from '../../src/detect/types'; +import { DetectRunStatus, ObjectEntityType, DeidentifyFileDataFormat } from '../../src/detect/types'; import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType, TestDetectController } from './test-types'; import { LogLevel } from '../../src/utils'; import SkyflowError from '../../src/error'; @@ -116,7 +116,7 @@ describe('output writer failures', () => { let thrown; try { await writeOutputs( - [{ processedFile: Buffer.from('a').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }], + [{ processedFile: Buffer.from('a').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: DeidentifyFileDataFormat.TXT }], DeidentifySourceType.BASE64, 'notes.txt', tmpDir, @@ -183,7 +183,7 @@ describe('output writer directory defaults', () => { process.chdir(tmpDir); try { const written = await writeOutputs( - [{ processedFile: Buffer.from('a').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }], + [{ processedFile: Buffer.from('a').toString('base64'), processedFileType: 'REDACTED_FILE', processedFileExtension: DeidentifyFileDataFormat.TXT }], DeidentifySourceType.BASE64, 'notes.txt', ' ', diff --git a/common/test/detect/file-format-enums.test.js b/common/test/detect/file-format-enums.test.js new file mode 100644 index 00000000..248ee6e0 --- /dev/null +++ b/common/test/detect/file-format-enums.test.js @@ -0,0 +1,45 @@ +import { DeidentifyFileDataFormat, ReidentifyFileDataFormat } from '../../src/detect/types'; +import { DeidentifyFileRequestV2DataFormat } from '../../src/_generated_/rest/api/resources/filesV2/types/DeidentifyFileRequestV2DataFormat'; +import { ReidentifyFileRequestV2DataFormat } from '../../src/_generated_/rest/api/resources/filesV2/types/ReidentifyFileRequestV2DataFormat'; +import { FileOutputProcessedFileExtension } from '../../src/_generated_/rest/api/types/FileOutputProcessedFileExtension'; +import { toFileOutput, toReidentifyFileOutput } from '../../src/detect/internal/response-mapper'; + +// The public file-format enums must mirror the generated wire enums value for value, so a caller can never +// select a format the endpoint rejects and never misses one it accepts. +const wire = (constObject) => Object.values(constObject).map((v) => String(v).toUpperCase()).sort(); +const sdk = (enumObject) => Object.values(enumObject).sort(); + +describe('Detect file-format enums', () => { + test('DeidentifyFileDataFormat mirrors the de-identify request wire enum', () => { + expect(sdk(DeidentifyFileDataFormat)).toEqual(wire(DeidentifyFileRequestV2DataFormat)); + }); + + test('DeidentifyFileDataFormat also mirrors the output extension wire enum', () => { + expect(sdk(DeidentifyFileDataFormat)).toEqual(wire(FileOutputProcessedFileExtension)); + }); + + test('ReidentifyFileDataFormat mirrors the re-identify request wire enum and has 8 values', () => { + expect(sdk(ReidentifyFileDataFormat)).toEqual(wire(ReidentifyFileRequestV2DataFormat)); + expect(Object.keys(ReidentifyFileDataFormat)).toHaveLength(8); + }); + + test('re-identify formats are a subset of de-identify formats', () => { + const deidentify = new Set(sdk(DeidentifyFileDataFormat)); + for (const f of sdk(ReidentifyFileDataFormat)) { + expect(deidentify.has(f)).toBe(true); + } + }); + + test('toReidentifyFileOutput narrows the extension to ReidentifyFileDataFormat', () => { + expect(toReidentifyFileOutput({ processedFile: 'b64', processedFileType: 'REIDENTIFIED_FILE', processedFileExtension: 'docx' })).toEqual({ + processedFile: 'b64', + processedFileType: 'REIDENTIFIED_FILE', + processedFileExtension: ReidentifyFileDataFormat.DOCX, + }); + // a format outside the re-identify set, or an unknown one, maps to null + expect(toReidentifyFileOutput({ processedFileExtension: 'pdf' }).processedFileExtension).toBeNull(); + expect(toReidentifyFileOutput({ processedFileExtension: 'weird' }).processedFileExtension).toBeNull(); + // the de-identify mapper still accepts the wider set + expect(toFileOutput({ processedFileExtension: 'pdf' }).processedFileExtension).toBe(DeidentifyFileDataFormat.PDF); + }); +}); diff --git a/common/test/detect/file-output-writer.test.js b/common/test/detect/file-output-writer.test.js index 4878212c..9342b447 100644 --- a/common/test/detect/file-output-writer.test.js +++ b/common/test/detect/file-output-writer.test.js @@ -2,7 +2,7 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import { writeOutputs, outputFileName, stripExtension } from '../../src/detect/internal/file-output-writer'; -import { FileDataFormat } from '../../src/detect/types'; +import { DeidentifyFileDataFormat } from '../../src/detect/types'; import { TestDeidentifySourceType as DeidentifySourceType } from './test-types'; import SkyflowError from '../../src/error'; @@ -27,11 +27,11 @@ describe('stripExtension', () => { describe('outputFileName', () => { test('names processed files and entity files, and avoids collisions', () => { const used = new Set(); - expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.PDF }, used)).toBe('processed-notes.pdf'); - expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'ENTITIES', processedFileExtension: FileDataFormat.JSON }, used)).toBe('processed-entities-notes.json'); + expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'REDACTED_FILE', processedFileExtension: DeidentifyFileDataFormat.PDF }, used)).toBe('processed-notes.pdf'); + expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'ENTITIES', processedFileExtension: DeidentifyFileDataFormat.JSON }, used)).toBe('processed-entities-notes.json'); expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'OBJECT_ENTITIES', processedFileExtension: null }, used)).toBe('processed-object-entities-notes.json'); // Same stem + extension again -> type inserted before the extension. - expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'REDACTED_TEXT', processedFileExtension: FileDataFormat.PDF }, used)).toBe('processed-notes.redacted_text.pdf'); + expect(outputFileName('notes', { processedFile: 'x', processedFileType: 'REDACTED_TEXT', processedFileExtension: DeidentifyFileDataFormat.PDF }, used)).toBe('processed-notes.redacted_text.pdf'); // No type and no extension -> .bin, and a counter on collision. expect(outputFileName('notes', { processedFile: 'x', processedFileType: null, processedFileExtension: null }, used)).toBe('processed-notes.bin'); expect(outputFileName('notes', { processedFile: 'x', processedFileType: null, processedFileExtension: null }, used)).toBe('processed-notes.5.bin'); @@ -49,9 +49,9 @@ describe('writeOutputs', () => { const dir = path.join(tmpDir, 'nested', 'out'); const written = await writeOutputs( [ - { processedFile: b64('redacted'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }, - { processedFile: b64('[]'), processedFileType: 'ENTITIES', processedFileExtension: FileDataFormat.JSON }, - { processedFile: null, processedFileType: 'REDACTED_TEXT', processedFileExtension: FileDataFormat.TXT }, + { processedFile: b64('redacted'), processedFileType: 'REDACTED_FILE', processedFileExtension: DeidentifyFileDataFormat.TXT }, + { processedFile: b64('[]'), processedFileType: 'ENTITIES', processedFileExtension: DeidentifyFileDataFormat.JSON }, + { processedFile: null, processedFileType: 'REDACTED_TEXT', processedFileExtension: DeidentifyFileDataFormat.TXT }, ], DeidentifySourceType.BASE64, 'notes.txt', @@ -74,7 +74,7 @@ describe('writeOutputs', () => { const blocker = path.join(tmpDir, 'file-not-dir'); fs.writeFileSync(blocker, 'x'); await expect( - writeOutputs([{ processedFile: b64('a'), processedFileType: 'REDACTED_FILE', processedFileExtension: FileDataFormat.TXT }], DeidentifySourceType.BASE64, 'a.txt', path.join(blocker, 'sub')), + writeOutputs([{ processedFile: b64('a'), processedFileType: 'REDACTED_FILE', processedFileExtension: DeidentifyFileDataFormat.TXT }], DeidentifySourceType.BASE64, 'a.txt', path.join(blocker, 'sub')), ).rejects.toBeInstanceOf(SkyflowError); }); }); diff --git a/common/test/detect/models.test.js b/common/test/detect/models.test.js index f73355ef..1e6ce552 100644 --- a/common/test/detect/models.test.js +++ b/common/test/detect/models.test.js @@ -13,7 +13,8 @@ import { CheckGuardrailsResponse, DetectRunStatus, EntityType, - FileDataFormat, + DeidentifyFileDataFormat, + ReidentifyFileDataFormat, GuardrailsValidation, POLL_OPTIONS, BaseDetectController, @@ -42,14 +43,14 @@ describe('detect request models', () => { const r = new DeidentifyFileRequest({ dataSource: DeidentifySourceType.PRESIGNED_URL, value: 'https://u', - dataFormat: FileDataFormat.PDF, + dataFormat: DeidentifyFileDataFormat.PDF, configurationId: 'c', pollOptions: { waitTime: 10 }, outputDirectory: '/out', }); expect(r.dataSource).toBe(DeidentifySourceType.PRESIGNED_URL); expect(r.value).toBe('https://u'); - expect(r.dataFormat).toBe(FileDataFormat.PDF); + expect(r.dataFormat).toBe(DeidentifyFileDataFormat.PDF); expect(r.configurationId).toBe('c'); expect(r.pollOptions).toEqual({ waitTime: 10 }); expect(r.outputDirectory).toBe('/out'); @@ -76,10 +77,10 @@ describe('detect request models', () => { expect(s.text).toBe('x'); expect(s.redactionLevel).toBeUndefined(); - const f = new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'v', { dataFormat: FileDataFormat.TXT }); + const f = new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'v', { dataFormat: ReidentifyFileDataFormat.TXT }); expect(f.dataSource).toBe(ReidentifySourceType.BASE64); expect(f.value).toBe('v'); - expect(f.dataFormat).toBe(FileDataFormat.TXT); + expect(f.dataFormat).toBe(ReidentifyFileDataFormat.TXT); expect(f.redactionLevel).toBeUndefined(); f.dataSource = ReidentifySourceType.PRESIGNED_URL; f.value = 'w'; diff --git a/common/test/detect/request-mapper.test.js b/common/test/detect/request-mapper.test.js index a2ddcf2a..478782c7 100644 --- a/common/test/detect/request-mapper.test.js +++ b/common/test/detect/request-mapper.test.js @@ -18,18 +18,7 @@ import DeidentifyFileRequest from '../../src/detect/model/request/deidentify-fil import ReidentifyStringRequest from '../../src/detect/model/request/reidentify-string'; import ReidentifyFileRequest from '../../src/detect/model/request/reidentify-file'; import CheckGuardrailsRequest from '../../src/detect/model/request/check-guardrails'; -import { - DeidentificationType, - EntityType, - FileDataFormat, - IncludePrefix, - MaskingMethod, - ObjectDeidentificationType, - ObjectEntityType, - OutputTranscriptionType, - PdfProcessingMode, - ReturnEntitiesType, -} from '../../src/detect/types'; +import { DeidentificationType, EntityType, ReidentifyFileDataFormat, IncludePrefix, MaskingMethod, ObjectDeidentificationType, ObjectEntityType, OutputTranscriptionType, PdfProcessingMode, ReturnEntitiesType, DeidentifyFileDataFormat } from '../../src/detect/types'; import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType } from './test-types'; import { RedactionType } from '../../src/utils'; import SkyflowError from '../../src/error'; @@ -209,7 +198,7 @@ describe('toDeidentifyFileRequest', () => { new DeidentifyFileRequest({ dataSource: DeidentifySourceType.SKYFLOW_ID, value: 'sid', - dataFormat: FileDataFormat.DOCX, + dataFormat: DeidentifyFileDataFormat.DOCX, configurationId: 'cfg', }), VAULT, @@ -226,7 +215,7 @@ describe('toDeidentifyFileRequest', () => { test('File request is encoded from arrayBuffer and an explicit dataFormat wins over inference', async () => { const file = { name: 'x.txt', arrayBuffer: async () => Uint8Array.from(Buffer.from('hello')).buffer }; const out = await toDeidentifyFileRequest( - new DeidentifyFileRequest({ file, dataFormat: FileDataFormat.CSV, dataSource: DeidentifySourceType.PRESIGNED_URL }), + new DeidentifyFileRequest({ file, dataFormat: DeidentifyFileDataFormat.CSV, dataSource: DeidentifySourceType.PRESIGNED_URL }), VAULT, ); expect(out.dataSource).toBe('BASE64'); // local file always wins @@ -254,8 +243,8 @@ describe('toDeidentifyFileRequest', () => { describe('inferFormat / localFileName', () => { test('inferFormat maps known extensions case-insensitively and returns undefined otherwise', () => { - expect(inferFormat('a.PDF')).toBe(FileDataFormat.PDF); - expect(inferFormat('a.jsonl')).toBe(FileDataFormat.JSONL); + expect(inferFormat('a.PDF')).toBe(DeidentifyFileDataFormat.PDF); + expect(inferFormat('a.jsonl')).toBe(DeidentifyFileDataFormat.JSONL); expect(inferFormat('a.unknown')).toBeUndefined(); expect(inferFormat('noext')).toBeUndefined(); expect(inferFormat(null)).toBeUndefined(); @@ -293,7 +282,7 @@ describe('reidentify / guardrails mappers', () => { test('toReidentifyFileRequest lower-cases the format and maps redaction levels', () => { const out = toReidentifyFileRequest( new ReidentifyFileRequest(ReidentifySourceType.BASE64, 'abc', { - dataFormat: FileDataFormat.TXT, + dataFormat: ReidentifyFileDataFormat.TXT, redactionLevel: [{ tokenGroupName: 'g' }], }), VAULT, diff --git a/common/test/detect/response-mapper.test.js b/common/test/detect/response-mapper.test.js index 56d7c919..8a1b503b 100644 --- a/common/test/detect/response-mapper.test.js +++ b/common/test/detect/response-mapper.test.js @@ -10,7 +10,7 @@ import { toReidentifyFileResponse as mapReidentifyFileResponse, toCheckGuardrailsResponse, } from '../../src/detect/internal/response-mapper'; -import { DetectRunStatus, FileDataFormat, GuardrailsValidation } from '../../src/detect/types'; +import { DetectRunStatus, GuardrailsValidation, DeidentifyFileDataFormat } from '../../src/detect/types'; import { TestDeidentifySourceType as DeidentifySourceType, TestReidentifySourceType as ReidentifySourceType, resolveDeidentifySource, resolveReidentifySource } from './test-types'; // The mappers take the SDK's source resolver; these bind the test stand-ins. @@ -83,11 +83,11 @@ describe('string responses', () => { }); describe('file responses', () => { - test('toFileOutput upper-cases the extension into FileDataFormat and tolerates unknowns', () => { + test('toFileOutput upper-cases the extension into DeidentifyFileDataFormat and tolerates unknowns', () => { expect(toFileOutput({ processedFile: 'b64', processedFileType: 'REDACTED_FILE', processedFileExtension: 'pdf' })).toEqual({ processedFile: 'b64', processedFileType: 'REDACTED_FILE', - processedFileExtension: FileDataFormat.PDF, + processedFileExtension: DeidentifyFileDataFormat.PDF, }); expect(toFileOutput({ processedFileExtension: 'weird' })).toEqual({ processedFile: null, diff --git a/docs/api_reference.md b/docs/api_reference.md index 21753ac7..e88f8e1a 100644 --- a/docs/api_reference.md +++ b/docs/api_reference.md @@ -415,7 +415,7 @@ Request classes take their required fields as constructor arguments and their op | `value` | `string` | Base64 content, a skyflow id, or a presigned URL, per `dataSource`. | | `file` | `File` | In-memory `File`; read and base64-encoded when sent, `dataSource` set to `BASE64`. Requires Node.js v20+. | | `filePath` | `string` | Path to a local file; same handling as `file`. | -| `dataFormat` | [`FileDataFormat`](#detect-enums) | File format. Inferred from the file name's extension when omitted for a local file. | +| `dataFormat` | [`DeidentifyFileDataFormat`](#detect-enums) | File format. Inferred from the file name's extension when omitted for a local file. | | `configurationId` | `string` | Id of a stored Detect configuration. Sending it together with `configuration` is rejected by the API. | | `configuration` | [`DetectConfiguration`](#detectconfiguration) | Inline configuration. With neither this nor `configurationId` set, the server applies the vault's defaults. | | `pollOptions` | [`PollOptions`](#polloptions) | Poll the run after submitting. Omit to submit only and return the run id. | @@ -438,7 +438,7 @@ Request classes take their required fields as constructor arguments and their op |---|---|---| | `dataSource` | [`ReidentifySourceType`](#detect-enums) | **Required.** Your SDK's enum: `BASE64` or `PRESIGNED_URL`. Re-identification does not accept a skyflow id as the source. | | `value` | `string` | **Required.** Base64 content or a presigned URL, per `dataSource`. | -| `dataFormat` | [`FileDataFormat`](#detect-enums) | Limited to the formats that support `VAULT_TOKEN`: `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` or `PPTX`. | +| `dataFormat` | [`ReidentifyFileDataFormat`](#detect-enums) | Limited to the formats that support `VAULT_TOKEN`: `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` or `PPTX`. | | `redactionLevel` | [`RedactionLevel[]`](#redactionlevel) | Redaction rules. | ### `GetRunRequest` @@ -499,7 +499,7 @@ Returned by `detect().reidentifyFile()`. |---|---|---| | `status` | [`DetectRunStatus`](#detect-enums) | Status of the re-identification. | | `outputType` | [`ReidentifySourceType`](#detect-enums)` \| null` | The output type resolved onto your SDK's enum; `null` when the value is not one the SDK offers. | -| `output` | [`FileOutput[]`](#fileoutput) | Re-identified file outputs, never `null`. | +| `output` | `ReidentifyFileOutput[]` | Re-identified file outputs, never `null`. Same fields as [`FileOutput`](#fileoutput), with `processedFileExtension` typed `ReidentifyFileDataFormat`. | | `metrics` | [`Metrics`](#metrics)` \| null` | Size and counts. | ### `GetRunResponse` @@ -660,7 +660,7 @@ Element of `output` on the file responses. |---|---|---| | `processedFile` | `string \| null` | Base64 content or a presigned URL, per the response's `outputType`. | | `processedFileType` | `string \| null` | `REDACTED_FILE`, `ENTITIES`, `OBJECT_ENTITIES`, `REDACTED_TEXT`, and so on. | -| `processedFileExtension` | `FileDataFormat \| null` | Extension of the output. | +| `processedFileExtension` | `DeidentifyFileDataFormat \| null` | Extension of the output. On `ReidentifyFileOutput` the same field is a `ReidentifyFileDataFormat`. | ### Detect enums @@ -673,7 +673,8 @@ All string enums whose values match the wire values. | `IncludePrefix` | `YES` _(server default)_, `NO`, `INCLUDE_PREFIX_UNSPECIFIED` _(field omitted)_. `VAULT_TOKEN` only. | | `DeidentifySourceType` | Per SDK: skyvault `BASE64`, `SKYFLOW_ID`, `PRESIGNED_URL`; flowvault `BASE64`, `PRESIGNED_URL`. Input source for `deidentifyFile`, and the `outputType` of `DeidentifyFileResponse` and `GetRunResponse`. | | `ReidentifySourceType` | Per SDK, `BASE64`, `PRESIGNED_URL` on both. Input source for `reidentifyFile` and the `outputType` of `ReidentifyFileResponse`. | -| `FileDataFormat` | `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML`, `ZIP` | +| `DeidentifyFileDataFormat` | `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML`, `ZIP`. The `dataFormat` of `deidentifyFile` (inferred from a local file's extension when omitted) and the `processedFileExtension` of `FileOutput` on de-identify and get-run responses. | +| `ReidentifyFileDataFormat` | `CSV`, `DOCX`, `JSON`, `JSONL`, `PPTX`, `TXT`, `XLSX`, `XML`. The `dataFormat` of `reidentifyFile` (only the formats that support `VAULT_TOKEN`) and the `processedFileExtension` of `ReidentifyFileOutput` on its response. | | `DetectRunStatus` | `QUEUED`, `IN_PROGRESS`, `SUCCESS`, `FAILED`, `UNKNOWN`. `SUCCESS`, `FAILED` and `UNKNOWN` end polling. | | `ReturnEntitiesType` | `EXCLUDE_SENSITIVE_DATA` _(server default)_, `ALL`, `NONE` | | `MaskingMethod` | `NONE`, `BLUR`, `BLACKBOX` | diff --git a/docs/migrate_detect.md b/docs/migrate_detect.md index 9a64d68f..3ed970c7 100644 --- a/docs/migrate_detect.md +++ b/docs/migrate_detect.md @@ -268,7 +268,7 @@ before.file; // File, first output only before.entities; // [{ file, extension }] before.pageCount; -// Replacement: same behaviour, opt in to polling +// Replacement: same behavior, opt in to polling const after = await skyflowClient.detect().deidentifyFile( new DeidentifyFileRequest({ filePath: '/data/intake/patient-notes.pdf', @@ -316,19 +316,20 @@ const runId = submitted.runId; // status is null, nothing written to | `errors` | Removed. Failures reject with `SkyflowError`; a run that failed while processing has `status: FAILED` and `message` | | — | `outputType`, `message` | -### Behaviour differences to check +### Behavior differences to check - **Polling is opt in.** Without `pollOptions` the call resolves with the run id immediately. Callers that relied on the result being complete must add `pollOptions`. - **Budget limits changed.** `waitTime` defaults to 60 and tops out at 300, where the old method defaulted to 64 and capped at 64. `maxAttempts` is new, default and maximum 23. -- **Elapsed time is measured by the clock.** The old loop counted only its planned delays. The new one measures real - elapsed time and never exceeds `waitTime`. +- **The budget counts the pauses between lookups.** Like the old loop, the time spent inside each status call is not + charged to `waitTime`, so wall-clock time can exceed it by the latency of the lookups. The final pause is trimmed so + the pauses alone never exceed `waitTime`. - **Output file names.** The redacted file is `processed-.` and the entities file is `processed-entities-.json`, written under `outputDirectory` (or the working directory when none is set) only for local file inputs that reached `SUCCESS` with base64 outputs. - **All outputs are returned.** The old response surfaced only the first output. `output` holds the full list. -- **`UNKNOWN` ends polling.** The old method kept calling the run endpoint on any status it did not recognise. The new one +- **`UNKNOWN` ends polling.** The old method kept calling the run endpoint on any status it did not recognize. The new one stops and returns `UNKNOWN`; check for it and call `getRun` later. - **Validation is stricter.** `waitTime` and `maxAttempts` out of range, `outputDirectory` without `pollOptions`, and supplying more than one of `file`, `filePath` and `value` reject before any network call. diff --git a/flowvault/README.md b/flowvault/README.md index b3538525..f8e395bd 100644 --- a/flowvault/README.md +++ b/flowvault/README.md @@ -437,7 +437,7 @@ File de-identification is asynchronous on the API: the submit call returns a run The file is supplied in one of three ways: a `dataSource` plus `value` (`DeidentifySourceType.BASE64` content or a `DeidentifySourceType.PRESIGNED_URL`; flowvault vaults do not take a skyflow id as a source), a `file` (a web `File`, Node.js 20+), or a `filePath`. A local file is read, base64-encoded, and its `dataFormat` inferred from the extension. Supply exactly one of `value`, `file` or `filePath`. `configurationId` and `configuration` follow the same rules as for strings. -`dataFormat` accepts every `FileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. +`dataFormat` accepts every `DeidentifyFileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. `outputDirectory` is meaningful only when polling. Setting it without `pollOptions` fails client-side validation before any request is sent. @@ -575,12 +575,12 @@ console.log(rendered.processedText); ```typescript import * as fs from 'fs'; -import { ReidentifyFileRequest, ReidentifySourceType, FileDataFormat, DetectRunStatus } from 'skyflow-flowvault-node'; +import { ReidentifyFileRequest, ReidentifySourceType, ReidentifyFileDataFormat, DetectRunStatus } from 'skyflow-flowvault-node'; const content = fs.readFileSync('tokenized.txt').toString('base64'); const response = await skyflowClient.detect().reidentifyFile( - new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: FileDataFormat.TXT }), + new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: ReidentifyFileDataFormat.TXT }), ); if (response.status === DetectRunStatus.SUCCESS) { @@ -625,7 +625,7 @@ switch (run.status) { case DetectRunStatus.SUCCESS: for (const output of run.output) { // output.processedFileType is e.g. REDACTED_FILE or ENTITIES - // output.processedFileExtension is a FileDataFormat such as PDF + // output.processedFileExtension is a DeidentifyFileDataFormat such as PDF if (run.outputType === DeidentifySourceType.BASE64) { const bytes = Buffer.from(output.processedFile ?? '', 'base64'); fs.writeFileSync(`processed-${output.processedFileType}.${output.processedFileExtension?.toLowerCase()}`, bytes); diff --git a/flowvault/api-report/skyflow-flowvault-node.api.md b/flowvault/api-report/skyflow-flowvault-node.api.md index 0685496f..d6eaadaf 100644 --- a/flowvault/api-report/skyflow-flowvault-node.api.md +++ b/flowvault/api-report/skyflow-flowvault-node.api.md @@ -4,35 +4,291 @@ ```ts -import { AuthInfo } from 'skyflow-common'; -import { BaseDeleteFilesRequest } from 'skyflow-common'; -import { BaseDeleteFilesResponse } from 'skyflow-common'; -import { BaseDeleteRequest } from 'skyflow-common'; -import { BaseDeleteResponse } from 'skyflow-common'; -import { BaseDetectController } from 'skyflow-common'; -import { BaseDetokenizeRequest } from 'skyflow-common'; -import { BaseDetokenizeResponse } from 'skyflow-common'; -import { BaseGetRequest } from 'skyflow-common'; -import { BaseGetResponse } from 'skyflow-common'; -import { BaseGetTokensRequest } from 'skyflow-common'; -import { BaseGetTokensResponse } from 'skyflow-common'; -import { BaseInsertRequest } from 'skyflow-common'; -import { BaseInsertResponse } from 'skyflow-common'; -import { BaseQueryRequest } from 'skyflow-common'; -import { BaseQueryResponse } from 'skyflow-common'; -import { BaseSkyflow } from 'skyflow-common'; -import { BaseUpdateRequest } from 'skyflow-common'; -import { BaseUpdateResponse } from 'skyflow-common'; -import { BaseUploadFilesRequest } from 'skyflow-common'; -import { BaseUploadFilesResponse } from 'skyflow-common'; -import { BaseVaultEntry } from 'skyflow-common'; -import { Credentials } from 'skyflow-common'; -import { LogLevel } from 'skyflow-common'; -import { RedactionRule } from 'skyflow-common'; -import { SkyflowConfig } from 'skyflow-common'; -import { SkyflowError } from 'skyflow-common'; -import { SkyflowRecordError } from 'skyflow-common'; -import { VaultConfig } from 'skyflow-common'; +// @public (undocumented) +export const API_KEY: { + readonly PREFIX: "sky-"; +}; + +// @public (undocumented) +export interface ApiKeyCredentials { + // (undocumented) + apiKey: string; +} + +// @public +interface Audio_2 { + // (undocumented) + bleep?: Bleep; + // (undocumented) + outputProcessedAudio?: boolean; + // (undocumented) + outputTranscription?: OutputTranscriptionType; +} +export { Audio_2 as Audio } + +// @public (undocumented) +export interface AuthInfo { + // (undocumented) + key: string; + // (undocumented) + type: AuthType; +} + +// @public (undocumented) +export enum AuthType { + // (undocumented) + API_KEY = "API_KEY", + // (undocumented) + TOKEN = "TOKEN" +} + +// @public (undocumented) +export const BAD_REQUEST = "Bad Request"; + +// @public (undocumented) +export class BaseDeleteFilesRequest { +} + +// @public (undocumented) +export class BaseDeleteFilesResponse { +} + +// @public (undocumented) +export class BaseDeleteRequest { +} + +// @public (undocumented) +export class BaseDeleteResponse { +} + +// @public (undocumented) +export abstract class BaseDetectController { + constructor(client: TClient); + checkGuardrails(request: CheckGuardrailsRequest): Promise; + // (undocumented) + protected readonly client: TClient; + deidentifyFile(request: DeidentifyFileRequest): Promise; + deidentifyString(request: DeidentifyStringRequest): Promise; + protected static enumOrNull(values: Record, value: string | null | undefined): E | null; + // Warning: (ae-forgotten-export) The symbol "FilesV2" needs to be exported by the entry point index.d.ts + protected files(): FilesV2; + getRun(request: GetRunRequest): Promise; + // Warning: (ae-forgotten-export) The symbol "GuardrailsV2" needs to be exported by the entry point index.d.ts + protected guardrails(): GuardrailsV2; + // (undocumented) + protected get logLevel(): LogLevel; + reidentifyFile(request: ReidentifyFileRequest): Promise; + reidentifyString(request: ReidentifyStringRequest): Promise; + protected sleeper(): ((millis: number) => Promise) | undefined; + // Warning: (ae-forgotten-export) The symbol "StringsV2" needs to be exported by the entry point index.d.ts + protected strings(): StringsV2; + protected abstract toDeidentifySource(value: string | null | undefined): DeidentifyFileSource | null; + protected abstract toReidentifySource(value: string | null | undefined): ReidentifyFileSource | null; +} + +// @public (undocumented) +export class BaseDetokenizeRequest { +} + +// @public (undocumented) +export class BaseDetokenizeResponse { +} + +// @public (undocumented) +export class BaseFileUploadRequest { +} + +// @public (undocumented) +export class BaseFileUploadResponse { +} + +// @public (undocumented) +export class BaseGetRequest { +} + +// @public (undocumented) +export class BaseGetResponse { +} + +// @public (undocumented) +export class BaseGetTokensRequest { +} + +// @public (undocumented) +export class BaseGetTokensResponse { +} + +// @public (undocumented) +export class BaseInsertRequest { +} + +// @public (undocumented) +export class BaseInsertResponse { +} + +// @public (undocumented) +export class BaseInvokeConnectionRequest { +} + +// @public (undocumented) +export class BaseInvokeConnectionResponse { +} + +// @public (undocumented) +export class BaseOptions { + // (undocumented) + getRequestHeaders(): Record | undefined; + // (undocumented) + setRequestHeaders(h: Record): void; +} + +// @public (undocumented) +export class BaseQueryRequest { +} + +// @public (undocumented) +export class BaseQueryResponse { +} + +// @public (undocumented) +export abstract class BaseSkyflow { + // (undocumented) + addVaultConfig(config: VaultConfig): void; + protected applyCredentials(): void; + protected applyLogLevel(): void; + protected abstract buildVaultEntry(config: VaultConfig): TEntry; + // (undocumented) + protected commonCredentials?: Credentials; + // (undocumented) + getLogLevel(): LogLevel; + // (undocumented) + getSkyflowCredentials(): Credentials | undefined; + // (undocumented) + getVaultConfig(vaultId: string): VaultConfig; + protected initializeSkyflow(config: SkyflowConfig): void; + // (undocumented) + protected logLevel: LogLevel; + // (undocumented) + removeVaultConfig(vaultId: string): void; + protected resolveVaultEntry(vaultId?: string): TEntry; + // (undocumented) + setLogLevel(logLevel: LogLevel): void; + // (undocumented) + protected throwErrorForUnknownVaultId(id: string): void; + // (undocumented) + protected throwErrorIfVaultIdExists(id: string): void; + // (undocumented) + updateLogLevel(logLevel: LogLevel): this; + // (undocumented) + updateSkyflowCredentials(credentials: Credentials): void; + // (undocumented) + updateVaultConfig(config: VaultConfig): void; + // (undocumented) + protected vaultClients: { + [vaultId: string]: TEntry; + }; +} + +// @public (undocumented) +export class BaseTokenizeRequest { +} + +// @public (undocumented) +export class BaseTokenizeResponse { +} + +// @public (undocumented) +export class BaseUpdateRequest { +} + +// @public (undocumented) +export class BaseUpdateResponse { +} + +// @public (undocumented) +export class BaseUploadFilesRequest { +} + +// @public (undocumented) +export class BaseUploadFilesResponse { +} + +// @public +export interface BaseVaultClientLike { + // (undocumented) + setLogLevel(logLevel: LogLevel): void; + // (undocumented) + updateClientConfig(url: string, vaultId: string, individualCredentials?: Credentials, skyflowCredentials?: Credentials, logLevel?: LogLevel): void; + // (undocumented) + updateSkyflowCredentials(credentials?: Credentials): void; +} + +// @public +export interface BaseVaultEntry { + // (undocumented) + client: TClient; + // (undocumented) + config: VaultConfig; +} + +// @public (undocumented) +export type BearerTokenOptions = { + ctx?: string | Record; + roleIDs?: string[]; + roleIds?: string[]; + logLevel?: LogLevel; + tokenUri?: string; +}; + +// @public +export interface Bleep { + // (undocumented) + frequency?: number; + // (undocumented) + gain?: number; + // (undocumented) + startPadding?: number; + // (undocumented) + stopPadding?: number; +} + +// @public (undocumented) +export const BOOLEAN_STRING: { + readonly TRUE: "true"; +}; + +// @public +export class CheckGuardrailsRequest { + constructor(text: string, fields?: CheckGuardrailsRequestFields); + // (undocumented) + get checkToxicity(): boolean | undefined; + set checkToxicity(value: boolean | undefined); + // (undocumented) + get denyTopics(): string[] | undefined; + set denyTopics(value: string[] | undefined); + get text(): string; + set text(value: string); +} + +// @public +export interface CheckGuardrailsRequestFields { + checkToxicity?: boolean; + denyTopics?: string[]; +} + +// @public +export class CheckGuardrailsResponse { + constructor(input: { + text: string | null; + validation: GuardrailsValidation; + toxic: boolean | null; + deniedTopic: boolean | null; + }); + readonly deniedTopic: boolean | null; + readonly text: string | null; + readonly toxic: boolean | null; + readonly validation: GuardrailsValidation; +} // @public export interface ColumnRedaction { @@ -42,6 +298,47 @@ export interface ColumnRedaction { redaction: string; } +// @public (undocumented) +export const CONFIG: { + readonly LOGLEVEL: "loglevel"; + readonly CREDENTIALS: "credentials"; + readonly VAULT_ID: "vaultId"; + readonly CONNECTION_ID: "connectionId"; + readonly VAULT: "vault"; + readonly CONNECTION: "connection"; +}; + +// @public (undocumented) +export interface ConnectionConfig { + // (undocumented) + connectionId: string; + // (undocumented) + connectionUrl: string; + // (undocumented) + credentials?: Credentials; +} + +// @public (undocumented) +export const CONTENT_TYPE: { + readonly APPLICATION_JSON: "application/json"; + readonly APPLICATION_X_WWW_FORM_URLENCODED: "application/x-www-form-urlencoded"; + readonly TEXT_PLAIN: "text/plain"; + readonly MULTIPART_FORM_DATA: "multipart/form-data"; + readonly TEXT_XML: "text/xml"; + readonly APPLICATION_XML: "application/xml"; + readonly TEXT_HTML: "text/html"; +}; + +// @public (undocumented) +export const CONTROLLER_TYPES: { + DETECT: string; + VAULT: string; + CONNECTION: string; +}; + +// @public (undocumented) +export type Credentials = TokenCredentials | PathCredentials | StringCredentials | ApiKeyCredentials; + // @public export enum CustomHeaderKey { // (undocumented) @@ -52,25 +349,899 @@ export enum CustomHeaderKey { SKYFLOW_ACCOUNT_NAME = "x-skyflow-account-name" } +// @public +export interface CustomType { + deidentificationType?: DeidentificationType; + destination?: string; + includePrefix?: IncludePrefix; + // (undocumented) + label?: string; + // (undocumented) + match?: string[]; + prefixOverride?: string; +} + +// @public +export enum DeidentificationType { + // (undocumented) + ENTITY_ONLY = "ENTITY_ONLY", + // (undocumented) + ENTITY_UNIQUE_COUNTER = "ENTITY_UNIQUE_COUNTER", + // (undocumented) + MASK = "MASK", + // (undocumented) + UNKNOWN = "UNKNOWN", + // (undocumented) + VAULT_TOKEN = "VAULT_TOKEN" +} + +// @public +export enum DeidentifyFileDataFormat { + // (undocumented) + BMP = "BMP", + // (undocumented) + CSV = "CSV", + // (undocumented) + DCM = "DCM", + // (undocumented) + DOC = "DOC", + // (undocumented) + DOCX = "DOCX", + // (undocumented) + GIF = "GIF", + // (undocumented) + JPEG = "JPEG", + // (undocumented) + JPG = "JPG", + // (undocumented) + JSON = "JSON", + // (undocumented) + JSONL = "JSONL", + // (undocumented) + MP3 = "MP3", + // (undocumented) + PDF = "PDF", + // (undocumented) + PNG = "PNG", + // (undocumented) + PPT = "PPT", + // (undocumented) + PPTX = "PPTX", + // (undocumented) + TIF = "TIF", + // (undocumented) + TIFF = "TIFF", + // (undocumented) + TXT = "TXT", + // (undocumented) + WAV = "WAV", + // (undocumented) + XLS = "XLS", + // (undocumented) + XLSX = "XLSX", + // (undocumented) + XML = "XML", + // (undocumented) + ZIP = "ZIP" +} + +// @public +export class DeidentifyFileRequest { + constructor(fields?: DeidentifyFileRequestFields); + // (undocumented) + get configuration(): DetectConfiguration | undefined; + set configuration(value: DetectConfiguration | undefined); + // (undocumented) + get configurationId(): string | undefined; + set configurationId(value: string | undefined); + // (undocumented) + get dataFormat(): DeidentifyFileDataFormat | undefined; + set dataFormat(value: DeidentifyFileDataFormat | undefined); + // (undocumented) + get dataSource(): DeidentifyFileSource | undefined; + set dataSource(value: DeidentifyFileSource | undefined); + // (undocumented) + get file(): File | undefined; + set file(value: File | undefined); + // (undocumented) + get filePath(): string | undefined; + set filePath(value: string | undefined); + // (undocumented) + get outputDirectory(): string | undefined; + set outputDirectory(value: string | undefined); + // (undocumented) + get pollOptions(): PollOptions | undefined; + set pollOptions(value: PollOptions | undefined); + // (undocumented) + get value(): string | undefined; + set value(value: string | undefined); +} + +// @public +export interface DeidentifyFileRequestFields { + configuration?: DetectConfiguration; + configurationId?: string; + dataFormat?: DeidentifyFileDataFormat; + dataSource?: DeidentifyFileSource; + file?: File; + filePath?: string; + outputDirectory?: string; + pollOptions?: PollOptions; + value?: string; +} + +// @public +export class DeidentifyFileResponse implements DetectRunFields { + constructor(input: DetectRunFields); + // (undocumented) + readonly message: string | null; + // (undocumented) + readonly metrics: Metrics | null; + // (undocumented) + readonly output: FileOutput[]; + // (undocumented) + readonly outputType: DeidentifyFileSource | null; + // (undocumented) + readonly runId: string | null; + // (undocumented) + readonly status: DetectRunStatus | null; +} + +// @public +export type DeidentifyFileSource = string; + // @public export enum DeidentifySourceType { // (undocumented) - BASE64 = "BASE64", + BASE64 = "BASE64", + // (undocumented) + PRESIGNED_URL = "PRESIGNED_URL" +} + +// @public +export class DeidentifyStringRequest { + constructor(text: string, fields?: DeidentifyStringRequestFields); + // (undocumented) + get configuration(): DetectConfiguration | undefined; + set configuration(value: DetectConfiguration | undefined); + // (undocumented) + get configurationId(): string | undefined; + set configurationId(value: string | undefined); + get text(): string; + set text(value: string); +} + +// @public +export interface DeidentifyStringRequestFields { + configuration?: DetectConfiguration; + configurationId?: string; +} + +// @public +export class DeidentifyStringResponse { + constructor(input: { + processedText: string | null; + entities: DetectedEntity[]; + metrics: Metrics | null; + }); + readonly entities: DetectedEntity[]; + readonly metrics: Metrics | null; + readonly processedText: string | null; +} + +// @public +export class DeleteFilesOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export interface DeleteFilesRecord { + // (undocumented) + columns: string[]; + // (undocumented) + skyflowId?: string; + // (undocumented) + tableName: string; + // (undocumented) + uniqueValues?: Record[]; +} + +// @public +export class DeleteFilesRequest extends BaseDeleteFilesRequest { + constructor(records: DeleteFilesRecord[]); + // (undocumented) + get records(): DeleteFilesRecord[]; + set records(value: DeleteFilesRecord[]); +} + +// @public +export class DeleteFilesResponse extends BaseDeleteFilesResponse { + constructor(input: { + records: DeleteFilesResponseRecord[]; + }); + // (undocumented) + readonly records: DeleteFilesResponseRecord[]; +} + +// @public (undocumented) +export interface DeleteFilesResponseColumn { + // (undocumented) + column: string; + // (undocumented) + status: string | null; +} + +// @public (undocumented) +export interface DeleteFilesResponseRecord { + // (undocumented) + columns: DeleteFilesResponseColumn[] | null; + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; + // (undocumented) + tableName?: string | null; +} + +// @public +export class DeleteOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export class DeleteRequest extends BaseDeleteRequest { + constructor(table: string, ids: string[], uniqueValues?: Record[]); + // (undocumented) + get ids(): string[]; + set ids(value: string[]); + // (undocumented) + get table(): string; + set table(value: string); + // (undocumented) + get uniqueValues(): Record[] | undefined; + set uniqueValues(value: Record[] | undefined); +} + +// @public +export class DeleteResponse extends BaseDeleteResponse { + constructor(input: { + records: DeleteResponseRecord[]; + }); + // (undocumented) + readonly records: DeleteResponseRecord[]; +} + +// @public (undocumented) +export interface DeleteResponseRecord { + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + requestId?: string | null; + // (undocumented) + skyflowId?: string | null; +} + +// @public +export interface Detect { + // (undocumented) + customTypes?: CustomType[]; + // (undocumented) + entities?: Entity[]; + objectEntities?: ObjectEntity[]; + restrict?: string[]; + returnEntities?: ReturnEntitiesType; + skip?: string[]; +} + +// @public +export interface DetectClientLike { + // (undocumented) + getLogLevel(): LogLevel; + readonly maxRetries?: number; + resolveAuthToken(): Promise; + readonly timeout?: number; + readonly url: string; + readonly vaultId: string; +} + +// @public +export interface DetectConfiguration { + // (undocumented) + description?: string; + // (undocumented) + detect?: Detect; + // (undocumented) + fileMapping?: FileMapping[]; + // (undocumented) + media?: Media; + // (undocumented) + name?: string; +} + +// @public +export interface DetectedEntity { + entityScores: Record; + entityType: string | null; + // (undocumented) + location: EntityLocation | null; + token: string | null; + value: string | null; +} + +// @public +export interface DetectRunFields { + message: string | null; + metrics: Metrics | null; + output: FileOutput[]; + outputType: DeidentifyFileSource | null; + runId: string | null; + status: DetectRunStatus | null; +} + +// @public +export enum DetectRunStatus { + // (undocumented) + FAILED = "FAILED", + // (undocumented) + IN_PROGRESS = "IN_PROGRESS", + // (undocumented) + QUEUED = "QUEUED", + // (undocumented) + SUCCESS = "SUCCESS", + // (undocumented) + UNKNOWN = "UNKNOWN" +} + +// @public +export class DetokenizeOptions extends FlowVaultOptions { + constructor(); +} + +// @public +export class DetokenizeRequest extends BaseDetokenizeRequest { + constructor(tokens: string[], tokenGroupRedactions?: TokenGroupRedaction[]); + // (undocumented) + get tokenGroupRedactions(): TokenGroupRedaction[] | undefined; + set tokenGroupRedactions(value: TokenGroupRedaction[] | undefined); + // (undocumented) + get tokens(): string[]; + set tokens(value: string[]); +} + +// @public +export class DetokenizeResponse extends BaseDetokenizeResponse { + constructor(input: { + records: DetokenizeResponseRecord[]; + }); + // (undocumented) + readonly records: DetokenizeResponseRecord[]; +} + +// @public (undocumented) +export interface DetokenizeResponseRecord { + // (undocumented) + error?: string | null; + // (undocumented) + httpCode: number; + // (undocumented) + metadata?: { + skyflowId?: string | null; + tableName?: string | null; + } | null; + // (undocumented) + requestId?: string | null; + // (undocumented) + token: string; + // (undocumented) + tokenGroupName?: string | null; + // (undocumented) + value?: unknown; +} + +// @public +interface Document_2 { + // (undocumented) + pdf?: Pdf; +} +export { Document_2 as Document } + +// @public (undocumented) +export const ENCODING_TYPE: { + readonly UTF8: "utf8"; + readonly BASE64: "base64"; + readonly BINARY: "binary"; +}; + +// @public +export interface Entity { + deidentificationType?: DeidentificationType; + destination?: string; + entityType?: EntityType; + includePrefix?: IncludePrefix; + prefixOverride?: string; + // (undocumented) + transformation?: Transformation; +} + +// @public +export interface EntityLocation { + // (undocumented) + endIndex: number | null; + // (undocumented) + endIndexProcessed: number | null; + // (undocumented) + startIndex: number | null; + // (undocumented) + startIndexProcessed: number | null; +} + +// @public +export enum EntityType { + // (undocumented) + ACCOUNT_NUMBER = "ACCOUNT_NUMBER", + // (undocumented) + AGE = "AGE", + // (undocumented) + ALL = "ALL", + // (undocumented) + BANK_ACCOUNT = "BANK_ACCOUNT", + // (undocumented) + BLOOD_TYPE = "BLOOD_TYPE", + // (undocumented) + CONDITION = "CONDITION", + // (undocumented) + CORPORATE_ACTION = "CORPORATE_ACTION", + // (undocumented) + CREDIT_CARD = "CREDIT_CARD", + // (undocumented) + CREDIT_CARD_EXPIRATION = "CREDIT_CARD_EXPIRATION", + // (undocumented) + CVV = "CVV", + // (undocumented) + DATE = "DATE", + // (undocumented) + DATE_INTERVAL = "DATE_INTERVAL", + // (undocumented) + DAY = "DAY", + // (undocumented) + DOB = "DOB", + // (undocumented) + DOSE = "DOSE", + // (undocumented) + DRIVER_LICENSE = "DRIVER_LICENSE", + // (undocumented) + DRUG = "DRUG", + // (undocumented) + DURATION = "DURATION", + // (undocumented) + EFFECT = "EFFECT", + // (undocumented) + EMAIL_ADDRESS = "EMAIL_ADDRESS", + // (undocumented) + EVENT = "EVENT", + // (undocumented) + FILENAME = "FILENAME", + // (undocumented) + FINANCIAL_METRIC = "FINANCIAL_METRIC", + // (undocumented) + GENDER = "GENDER", + // (undocumented) + HEALTHCARE_NUMBER = "HEALTHCARE_NUMBER", + // (undocumented) + INJURY = "INJURY", + // (undocumented) + IP_ADDRESS = "IP_ADDRESS", + // (undocumented) + LANGUAGE = "LANGUAGE", + // (undocumented) + LOCATION = "LOCATION", + // (undocumented) + LOCATION_ADDRESS = "LOCATION_ADDRESS", + // (undocumented) + LOCATION_ADDRESS_STREET = "LOCATION_ADDRESS_STREET", + // (undocumented) + LOCATION_CITY = "LOCATION_CITY", + // (undocumented) + LOCATION_COORDINATE = "LOCATION_COORDINATE", + // (undocumented) + LOCATION_COUNTRY = "LOCATION_COUNTRY", + // (undocumented) + LOCATION_STATE = "LOCATION_STATE", + // (undocumented) + LOCATION_ZIP = "LOCATION_ZIP", + // (undocumented) + MARITAL_STATUS = "MARITAL_STATUS", + // (undocumented) + MEDICAL_CODE = "MEDICAL_CODE", + // (undocumented) + MEDICAL_PROCESS = "MEDICAL_PROCESS", + // (undocumented) + MONEY = "MONEY", + // (undocumented) + MONTH = "MONTH", + // (undocumented) + NAME = "NAME", + // (undocumented) + NAME_FAMILY = "NAME_FAMILY", + // (undocumented) + NAME_GIVEN = "NAME_GIVEN", + // (undocumented) + NAME_MEDICAL_PROFESSIONAL = "NAME_MEDICAL_PROFESSIONAL", + // (undocumented) + NONE = "NONE", + // (undocumented) + NUMERICAL_PII = "NUMERICAL_PII", + // (undocumented) + OCCUPATION = "OCCUPATION", + // (undocumented) + ORGANIZATION = "ORGANIZATION", + // (undocumented) + ORGANIZATION_ID = "ORGANIZATION_ID", + // (undocumented) + ORGANIZATION_MEDICAL_FACILITY = "ORGANIZATION_MEDICAL_FACILITY", + // (undocumented) + ORIGIN = "ORIGIN", + // (undocumented) + PASSPORT_NUMBER = "PASSPORT_NUMBER", + // (undocumented) + PASSWORD = "PASSWORD", + // (undocumented) + PHONE_NUMBER = "PHONE_NUMBER", + // (undocumented) + PHYSICAL_ATTRIBUTE = "PHYSICAL_ATTRIBUTE", + // (undocumented) + POLITICAL_AFFILIATION = "POLITICAL_AFFILIATION", + // (undocumented) + PRODUCT = "PRODUCT", + // (undocumented) + PROJECT = "PROJECT", + // (undocumented) + RELIGION = "RELIGION", + // (undocumented) + ROUTING_NUMBER = "ROUTING_NUMBER", + // (undocumented) + SEXUALITY = "SEXUALITY", + // (undocumented) + SSN = "SSN", + // (undocumented) + STATISTICS = "STATISTICS", + // (undocumented) + TIME = "TIME", + // (undocumented) + TREND = "TREND", + // (undocumented) + URL = "URL", + // (undocumented) + USERNAME = "USERNAME", + // (undocumented) + VEHICLE_ID = "VEHICLE_ID", + // (undocumented) + YEAR = "YEAR", + // (undocumented) + ZODIAC_SIGN = "ZODIAC_SIGN" +} + +// @public (undocumented) +export enum Env { + // (undocumented) + DEV = "DEV", + // (undocumented) + PROD = "PROD", + // (undocumented) + SANDBOX = "SANDBOX", + // (undocumented) + STAGE = "STAGE" +} + +// @public (undocumented) +export const errorMessages: { + readonly CONFIG_MISSING: string; + readonly INVALID_SKYFLOW_CONFIG: string; + readonly INVALID_TYPE_FOR_CONFIG: string; + readonly EMPTY_VAULT_CONFIG: string; + readonly EMPTY_CONNECTION_CONFIG: string; + readonly EMPTY_VAULT_ID: string; + readonly EMPTY_VAULT_ID_VALIDATION: string; + readonly INVALID_VAULT_ID: string; + readonly EMPTY_CLUSTER_ID: string; + readonly INVALID_CLUSTER_ID: string; + readonly INVALID_TOKEN: string; + readonly TOKEN_EXPIRED: string; + readonly INVALID_ENV: string; + readonly INVALID_LOG_LEVEL: string; + readonly EMPTY_CREDENTIAL_FILE_PATH: string; + readonly INVALID_CREDENTIAL_FILE_PATH: string; + readonly INVALID_CREDENTIALS_FILE_PATH: string; + readonly INVALID_TOKEN_URI: string; + readonly INVALID_TOKEN_URI_WITH_ID: string; + readonly INVALID_KEY: string; + readonly INVALID_PARSED_CREDENTIALS_STRING: string; + readonly INVALID_BEARER_TOKEN: string; + readonly INVALID_FILE_PATH_WITH_ID: string; + readonly INVALID_KEY_WITH_ID: string; + readonly INVALID_PARSED_CREDENTIALS_STRING_WITH_ID: string; + readonly INVALID_BEARER_TOKEN_WITH_ID: string; + readonly EMPTY_CONNECTION_ID_VALIDATION: string; + readonly EMPTY_CONNECTION_ID: string; + readonly INVALID_CONNECTION_ID: string; + readonly EMPTY_CONNECTION_URL: string; + readonly INVALID_CONNECTION_URL: string; + readonly VAULT_ID_EXITS_IN_CONFIG_LIST: string; + readonly CONNECTION_ID_EXITS_IN_CONFIG_LIST: string; + readonly VAULT_ID_NOT_IN_CONFIG_LIST: string; + readonly CONNECTION_ID_NOT_IN_CONFIG_LIST: string; + readonly EMPTY_CREDENTIALS: string; + readonly INVALID_CREDENTIALS: string; + readonly CREDENTIALS_WITH_NO_VALID_KEY: string; + readonly MULTIPLE_CREDENTIALS_PASSED: string; + readonly INVALID_CREDENTIALS_WITH_ID: string; + readonly MULTIPLE_CREDENTIALS_PASSED_WITH_ID: string; + readonly FILE_NOT_FOUND: string; + readonly INVALID_JSON_FILE: string; + readonly EMPTY_CREDENTIALS_STRING: string; + readonly INVALID_CREDENTIALS_STRING: string; + readonly MISSING_PRIVATE_KEY: string; + readonly MISSING_CLIENT_ID: string; + readonly MISSING_KEY_ID: string; + readonly MISSING_TOKEN_URI: string; + readonly INVALID_ROLES_KEY_TYPE: string; + readonly INVALID_CONTEXT: string; + readonly EMPTY_ROLES: string; + readonly INVALID_JSON_FORMAT: string; + readonly EMPTY_DATA_TOKENS: string; + readonly DATA_TOKEN_KEY_TYPE: string; + readonly TIME_TO_LIVE_KET_TYPE: string; + readonly INVALID_DELETE_IDS_INPUT: string; + readonly EMPTY_DELETE_IDS: string; + readonly INVALID_ID_IN_DELETE: string; + readonly INVALID_DELETE_REQUEST: string; + readonly EMPTY_ID_IN_DELETE: string; + readonly MISSING_REDACTION_TYPE_IN_DETOKENIZE: string; + readonly INVALID_REDACTION_TYPE_IN_DETOKENIZE: string; + readonly INVALID_TOKENS_TYPE_IN_DETOKENIZE: string; + readonly EMPTY_TOKENS_IN_DETOKENIZE: string; + readonly EMPTY_TOKEN_IN_DETOKENIZE: string; + readonly INVALID_TOKEN_IN_DETOKENIZE: string; + readonly INVALID_DETOKENIZE_REQUEST: string; + readonly MISSING_VALUES_IN_TOKENIZE: string; + readonly INVALID_VALUES_TYPE_IN_TOKENIZE: string; + readonly EMPTY_VALUES_IN_TOKENIZE: string; + readonly EMPTY_DATA_IN_TOKENIZE: string; + readonly INVALID_DATA_IN_TOKENIZE: string; + readonly INVALID_COLUMN_GROUP_IN_TOKENIZE: string; + readonly INVALID_VALUE_IN_TOKENIZE: string; + readonly INVALID_TOKENIZE_REQUEST: string; + readonly EMPTY_COLUMN_GROUP_IN_TOKENIZE: string; + readonly EMPTY_VALUE_IN_TOKENIZE: string; + readonly INVALID_RECORD_IN_INSERT: string; + readonly INVALID_RECORD_IN_UPDATE: string; + readonly EMPTY_RECORD_IN_INSERT: string; + readonly INVALID_INSERT_REQUEST: string; + readonly INVALID_TYPE_OF_DATA_IN_INSERT: string; + readonly EMPTY_DATA_IN_INSERT: string; + readonly INVALID_QUERY_REQUEST: string; + readonly EMPTY_QUERY: string; + readonly INVALID_QUERY: string; + readonly INVALID_FILE_UPLOAD_REQUEST: string; + readonly MISSING_TABLE_IN_UPLOAD_FILE: string; + readonly INVALID_TABLE_IN_UPLOAD_FILE: string; + readonly MISSING_SKYFLOW_ID_IN_UPLOAD_FILE: string; + readonly INVALID_SKYFLOW_ID_IN_UPLOAD_FILE: string; + readonly MISSING_COLUMN_NAME_IN_UPLOAD_FILE: string; + readonly INVALID_COLUMN_NAME_IN_UPLOAD_FILE: string; + readonly MISSING_FILE_PATH_IN_UPLOAD_FILE: string; + readonly INVALID_FILE_PATH_IN_UPLOAD_FILE: string; + readonly MISSING_FILE_SOURCE_IN_UPLOAD_FILE: string; + readonly MISSING_FILE_NAME_FOR_BASE64: string; + readonly INVALID_FILE_OBJECT_IN_UPLOAD_FILE: string; + readonly MISSING_FILE_NAME_IN_FILE_OBJECT: string; + readonly INVALID_BASE64_IN_UPLOAD_FILE: string; + readonly MISSING_SKYFLOW_ID_IN_UPDATE: string; + readonly INVALID_SKYFLOW_ID_IN_UPDATE: string; + readonly INVALID_TYPE_OF_UPDATE_DATA: string; + readonly EMPTY_UPDATE_DATA: string; + readonly INVALID_UPDATE_REQUEST: string; + readonly INVALID_DATA_IN_UPDATE: string; + readonly EMPTY_DATA_IN_UPDATE: string; + readonly INVALID_UPDATE_TOKENS: string; + readonly INVALID_TOKEN_IN_UPDATE: string; + readonly EMPTY_TABLE_NAME: string; + readonly INVALID_TABLE_NAME: string; + readonly EMPTY_REDACTION_TYPE: string; + readonly INVALID_REDACTION_TYPE: string; + readonly INVALID_TYPE_OF_IDS: string; + readonly EMPTY_IDS_IN_GET: string; + readonly EMPTY_ID_IN_GET: string; + readonly INVALID_ID_IN_GET: string; + readonly INVALID_GET_REQUEST: string; + readonly EMPTY_COLUMN_NAME: string; + readonly INVALID_COLUMN_NAME: string; + readonly INVALID_COLUMN_VALUES: string; + readonly EMPTY_COLUMN_VALUES: string; + readonly EMPTY_COLUMN_VALUE: string; + readonly INVALID_COLUMN_VALUE: string; + readonly INVALID_GET_COLUMN_REQUEST: string; + readonly EMPTY_URL: string; + readonly INVALID_URL: string; + readonly EMPTY_METHOD_NAME: string; + readonly INVALID_METHOD_NAME: string; + readonly EMPTY_PATH_PARAMS: string; + readonly INVALID_PATH_PARAMS: string; + readonly EMPTY_QUERY_PARAMS: string; + readonly INVALID_QUERY_PARAMS: string; + readonly EMPTY_BODY: string; + readonly INVALID_BODY: string; + readonly EMPTY_HEADERS: string; + readonly INVALID_HEADERS: string; + readonly INVALID_INVOKE_CONNECTION_REQUEST: string; + readonly ERROR_OCCURRED: string; + readonly EMPTY_INSERT_TOKEN: string; + readonly INVALID_INSERT_TOKEN: string; + readonly INVALID_INSERT_TOKENS: string; + readonly INVALID_TOKEN_MODE: string; + readonly INVALID_HOMOGENEOUS: string; + readonly INVALID_TOKEN_STRICT: string; + readonly INVALID_CONTINUE_ON_ERROR: string; + readonly INVALID_UPSERT: string; + readonly INVALID_RETURN_TOKEN: string; + readonly NO_TOKENS_WITH_TOKEN_MODE: string; + readonly INSUFFICIENT_TOKENS_PASSED_FOR_TOKEN_MODE_ENABLE_STRICT: string; + readonly INVALID_DOWNLOAD_URL: string; + readonly EMPTY_FIELD: string; + readonly INVALID_FIELD: string; + readonly INVALID_OFFSET: string; + readonly INVALID_LIMIT: string; + readonly INVALID_ORDER_BY: string; + readonly INVALID_FIELDS: string; + readonly INVAILD_JSON_RESPONSE: string; + readonly EMPTY_VAULT_CLIENTS: string; + readonly EMPTY_CONNECTION_CLIENTS: string; + readonly GENERIC_API_ERROR: string; + readonly EMPTY_FILE_OBJECT: string; + readonly INVALID_FILE_FORMAT: string; + readonly MISSING_FILE_SOURCE: string; + readonly INVALID_FILE_OBJECT: string; + readonly INVALID_BASE64_STRING: string; + readonly INVALID_FILE_TYPE: string; + readonly INVALID_FILE_NAME: string; + readonly FILE_READ_ERROR: string; + readonly INVALID_BASE64_HEADER: string; + readonly INVALID_DEIDENTIFY_STRING_REQUEST: string; + readonly INVALID_TEXT_IN_DEIDENTIFY_STRING: string; + readonly INVALID_REIDENTIFY_STRING_REQUEST: string; + readonly INVALID_TEXT_IN_REIDENTIFY_STRING: string; + readonly INVALID_REDACTION_LEVEL: string; + readonly INVALID_DEIDENTIFY_FILE_REQUEST: string; + readonly INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE: string; + readonly INVALID_VALUE_IN_DEIDENTIFY_FILE: string; + readonly MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE: string; + readonly MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE: string; + readonly INVALID_FILE_IN_DEIDENTIFY_FILE: string; + readonly FILE_NOT_FOUND_IN_DEIDENTIFY_FILE: string; + readonly FAILED_TO_ENCODE_FILE: string; + readonly INVALID_WAIT_TIME_IN_POLL_OPTIONS: string; + readonly INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS: string; + readonly OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS: string; + readonly FAILED_TO_WRITE_OUTPUT_FILE: string; + readonly INVALID_REIDENTIFY_FILE_REQUEST: string; + readonly INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE: string; + readonly INVALID_VALUE_IN_REIDENTIFY_FILE: string; + readonly INVALID_GET_RUN_REQUEST: string; + readonly INVALID_RUN_ID_IN_GET_RUN: string; + readonly INVALID_CHECK_GUARDRAILS_REQUEST: string; + readonly INVALID_TEXT_IN_CHECK_GUARDRAILS: string; + readonly INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS: string; + readonly INTERNAL_SERVER_ERROR: string; + readonly INVALID_XML_FORMAT: string; +}; + +// @public +export interface FileMapping { + // (undocumented) + destination?: string; + // (undocumented) + entitiesDestination?: string; + // (undocumented) + objectEntitiesDestination?: string; + // (undocumented) + outputFileNameSuffix?: string; + // (undocumented) + source?: string; +} + +// @public +export interface FileOutput { + processedFile: string | null; + // (undocumented) + processedFileExtension: DeidentifyFileDataFormat | null; + processedFileType: string | null; +} + +// @public (undocumented) +export function fillUrlWithPathAndQueryParams(url: string, pathParams?: StringKeyValueMapType, queryParams?: StringKeyValueMapType): string; + +// @public +export class FlowVaultOptions { + getInterceptor(): RequestInterceptor | undefined; + setInterceptor(interceptor: RequestInterceptor): void; +} + +// @public +export type FlowVaultRecordError = SkyflowRecordError; + +// @public (undocumented) +export function generateBearerToken(credentialsFilePath: string, options?: BearerTokenOptions): Promise; + +// @public (undocumented) +export function generateBearerTokenFromCreds(credentials: any, options?: BearerTokenOptions): Promise; + +// @public (undocumented) +export const generateSDKMetrics: (logLevel?: LogLevel) => { + sdk_name_version: string; + sdk_client_device_model: string; + sdk_client_os_details: string; + sdk_runtime_details: string; +}; + +// @public (undocumented) +export function generateSignedDataTokens(credentialsFilePath: string, options: SignedDataTokensOptions): Promise; + +// @public (undocumented) +export function generateSignedDataTokensFromCreds(credentials: any, options: SignedDataTokensOptions): Promise; + +// @public (undocumented) +export type GenerateTokenOptions = { + logLevel?: LogLevel; +}; + +// @public (undocumented) +export function getBaseUrl(url: string): string; + +// @public (undocumented) +export function getBearerToken(credentials?: Credentials, logLevel?: LogLevel): Promise; + +// @public (undocumented) +export function getConnectionBaseURL(clusterId: string, env: Env): string; + +// @public +export class GetOptions extends FlowVaultOptions { + constructor(); + // (undocumented) + getColumnRedactions(): ColumnRedaction[] | undefined; + // (undocumented) + getFields(): string[] | undefined; + // (undocumented) + getLimit(): number | undefined; + // (undocumented) + getOffset(): number | undefined; + // (undocumented) + getUniqueValues(): Record[] | undefined; + // (undocumented) + setColumnRedactions(columnRedactions: ColumnRedaction[]): void; + // (undocumented) + setFields(fields: string[]): void; // (undocumented) - PRESIGNED_URL = "PRESIGNED_URL" -} - -// @public -export class DeleteFilesOptions extends FlowVaultOptions { - constructor(); + setLimit(limit: number): void; + // (undocumented) + setOffset(offset: number): void; + // (undocumented) + setUniqueValues(uniqueValues: Record[]): void; } // @public -export interface DeleteFilesRecord { +export interface GetRecord { // (undocumented) - columns: string[]; + columnRedactions?: ColumnRedaction[]; // (undocumented) - skyflowId?: string; + columns?: string[]; + // (undocumented) + ids: string[]; // (undocumented) tableName: string; // (undocumented) @@ -78,37 +1249,43 @@ export interface DeleteFilesRecord { } // @public -export class DeleteFilesRequest extends BaseDeleteFilesRequest { - constructor(records: DeleteFilesRecord[]); +export class GetRequest extends BaseGetRequest { + constructor(table: string, ids: string[], records?: GetRecord[]); // (undocumented) - get records(): DeleteFilesRecord[]; - set records(value: DeleteFilesRecord[]); + get ids(): string[]; + set ids(value: string[]); + // (undocumented) + get records(): GetRecord[] | undefined; + set records(value: GetRecord[] | undefined); + // (undocumented) + get table(): string; + set table(value: string); } // @public -export class DeleteFilesResponse extends BaseDeleteFilesResponse { +export class GetResponse extends BaseGetResponse { constructor(input: { - records: DeleteFilesResponseRecord[]; + records: GetResponseRecord[]; }); // (undocumented) - readonly records: DeleteFilesResponseRecord[]; + readonly records: GetResponseRecord[]; } // @public (undocumented) -export interface DeleteFilesResponseColumn { - // (undocumented) - column: string; +export interface GetResponseData { // (undocumented) - status: string | null; + [key: string]: unknown; } // @public (undocumented) -export interface DeleteFilesResponseRecord { +export interface GetResponseRecord { // (undocumented) - columns: DeleteFilesResponseColumn[] | null; + data?: Record | null; // (undocumented) error?: string | null; // (undocumented) + hashedData?: Record | null; + // (undocumented) httpCode: number; // (undocumented) requestId?: string | null; @@ -116,167 +1293,179 @@ export interface DeleteFilesResponseRecord { skyflowId?: string | null; // (undocumented) tableName?: string | null; + // (undocumented) + tokens?: Record | null; } // @public -export class DeleteOptions extends FlowVaultOptions { - constructor(); +export class GetRunRequest { + constructor(runId: string); + get runId(): string; + set runId(value: string); } // @public -export class DeleteRequest extends BaseDeleteRequest { - constructor(table: string, ids: string[], uniqueValues?: Record[]); - // (undocumented) - get ids(): string[]; - set ids(value: string[]); - // (undocumented) - get table(): string; - set table(value: string); +export class GetRunResponse implements DetectRunFields { + constructor(input: DetectRunFields); // (undocumented) - get uniqueValues(): Record[] | undefined; - set uniqueValues(value: Record[] | undefined); -} - -// @public -export class DeleteResponse extends BaseDeleteResponse { - constructor(input: { - records: DeleteResponseRecord[]; - }); + readonly message: string | null; // (undocumented) - readonly records: DeleteResponseRecord[]; -} - -// @public (undocumented) -export interface DeleteResponseRecord { + readonly metrics: Metrics | null; // (undocumented) - error?: string | null; + readonly output: FileOutput[]; // (undocumented) - httpCode: number; + readonly outputType: DeidentifyFileSource | null; // (undocumented) - requestId?: string | null; + readonly runId: string | null; // (undocumented) - skyflowId?: string | null; + readonly status: DetectRunStatus | null; } +// @public (undocumented) +export function getSdkIdentity(): SdkIdentity; + +// @public (undocumented) +export function getToken(credentials: Credentials, logLevel?: LogLevel): Promise<{ + accessToken: string; +}>; + // @public -export class DetokenizeOptions extends FlowVaultOptions { +export class GetTokensOptions extends FlowVaultOptions { constructor(); } // @public -export class DetokenizeRequest extends BaseDetokenizeRequest { - constructor(tokens: string[], tokenGroupRedactions?: TokenGroupRedaction[]); +export interface GetTokensRecord { // (undocumented) - get tokenGroupRedactions(): TokenGroupRedaction[] | undefined; - set tokenGroupRedactions(value: TokenGroupRedaction[] | undefined); + tokenGroupName: string; // (undocumented) - get tokens(): string[]; - set tokens(value: string[]); + value: unknown; } // @public -export class DetokenizeResponse extends BaseDetokenizeResponse { +export class GetTokensRequest extends BaseGetTokensRequest { + constructor(records: GetTokensRecord[]); + // (undocumented) + get records(): GetTokensRecord[]; + set records(value: GetTokensRecord[]); +} + +// @public +export class GetTokensResponse extends BaseGetTokensResponse { constructor(input: { - records: DetokenizeResponseRecord[]; + records: GetTokensResponseRecord[]; }); // (undocumented) - readonly records: DetokenizeResponseRecord[]; + readonly records: GetTokensResponseRecord[]; } // @public (undocumented) -export interface DetokenizeResponseRecord { +export interface GetTokensResponseRecord { // (undocumented) error?: string | null; // (undocumented) httpCode: number; // (undocumented) - metadata?: { - skyflowId?: string | null; - tableName?: string | null; - } | null; - // (undocumented) requestId?: string | null; // (undocumented) - token: string; + token?: string | null; // (undocumented) - tokenGroupName?: string | null; + tokenGroupName: string | null; // (undocumented) - value?: unknown; -} - -// @public -export class FlowVaultOptions { - getInterceptor(): RequestInterceptor | undefined; - setInterceptor(interceptor: RequestInterceptor): void; + value: unknown; } -// @public -export type FlowVaultRecordError = SkyflowRecordError; +// @public (undocumented) +export function getVaultURL(clusterId: string, env: Env): string; // @public -export class GetOptions extends FlowVaultOptions { - constructor(); - // (undocumented) - getColumnRedactions(): ColumnRedaction[] | undefined; +export enum GuardrailsValidation { // (undocumented) - getFields(): string[] | undefined; + FAILED = "FAILED", // (undocumented) - getLimit(): number | undefined; + PASSED = "PASSED", // (undocumented) - getOffset(): number | undefined; + UNKNOWN = "UNKNOWN" +} + +// @public (undocumented) +export const HTTP_HEADER: { + readonly CONTENT_TYPE: "Content-Type"; + readonly CONTENT_TYPE_LOWER: string; + readonly X_REQUEST_ID: "x-request-id"; + readonly ERROR_FROM_CLIENT: "error-from-client"; +}; + +// @public (undocumented) +export const HTTP_STATUS_CODE: { + readonly OK: 200; + readonly BAD_REQUEST: 400; + readonly INTERNAL_SERVER_ERROR: 500; +}; + +// @public +interface Image_2 { // (undocumented) - getUniqueValues(): Record[] | undefined; + maskingMethod?: MaskingMethod; // (undocumented) - setColumnRedactions(columnRedactions: ColumnRedaction[]): void; + outputOcrText?: boolean; // (undocumented) - setFields(fields: string[]): void; + outputProcessedImage?: boolean; +} +export { Image_2 as Image } + +// @public +export enum IncludePrefix { // (undocumented) - setLimit(limit: number): void; + INCLUDE_PREFIX_UNSPECIFIED = "INCLUDE_PREFIX_UNSPECIFIED", // (undocumented) - setOffset(offset: number): void; + NO = "NO", // (undocumented) - setUniqueValues(uniqueValues: Record[]): void; + YES = "YES" } // @public -export interface GetRecord { +export class InsertOptions extends FlowVaultOptions { + constructor(); // (undocumented) - columnRedactions?: ColumnRedaction[]; + getUpsert(): UpsertOptions | undefined; // (undocumented) - columns?: string[]; + setUpsert(upsert: UpsertOptions): void; +} + +// @public +export interface InsertRecord { // (undocumented) - ids: string[]; + data: Record; // (undocumented) - tableName: string; + table?: string; // (undocumented) - uniqueValues?: Record[]; + tokens?: Record; + upsert?: UpsertOptions; } // @public -export class GetRequest extends BaseGetRequest { - constructor(table: string, ids: string[], records?: GetRecord[]); - // (undocumented) - get ids(): string[]; - set ids(value: string[]); +export class InsertRequest extends BaseInsertRequest { + constructor(tableName: string, records: InsertRecord[]); // (undocumented) - get records(): GetRecord[] | undefined; - set records(value: GetRecord[] | undefined); + get records(): InsertRecord[]; + set records(value: InsertRecord[]); // (undocumented) - get table(): string; - set table(value: string); + get tableName(): string; + set tableName(value: string); } // @public -export class GetResponse extends BaseGetResponse { +export class InsertResponse extends BaseInsertResponse { constructor(input: { - records: GetResponseRecord[]; + records: InsertResponseRecord[]; }); // (undocumented) - readonly records: GetResponseRecord[]; + readonly records: InsertResponseRecord[]; } // @public (undocumented) -export interface GetResponseRecord { +export interface InsertResponseRecord { // (undocumented) data?: Record | null; // (undocumented) @@ -295,112 +1484,497 @@ export interface GetResponseRecord { tokens?: Record | null; } +// @public (undocumented) +export interface InsertResponseType { + // (undocumented) + [key: string]: unknown; + // @deprecated (undocumented) + skyflow_id?: string; + // (undocumented) + skyflowId: string; +} + +// @public (undocumented) +export function isEnv(value?: string): boolean; + +// @public (undocumented) +export function isExpired(token: string): boolean; + +// @public (undocumented) +export interface ISkyflowError { + // (undocumented) + details?: Array | null; + // @deprecated (undocumented) + grpc_code?: string | number | null; + // (undocumented) + grpcCode?: string | number | null; + // @deprecated (undocumented) + http_code?: string | number | null | undefined; + // @deprecated (undocumented) + http_status?: string | number | null; + // (undocumented) + httpCode?: string | number | null | undefined; + // (undocumented) + httpStatus?: string | number | null; + // (undocumented) + message: string; + // @deprecated (undocumented) + request_ID?: string | null; + // (undocumented) + requestId?: string | null; +} + +// @public (undocumented) +export function isLogLevel(value?: string): boolean; + +// @public (undocumented) +export function isTokenValid(token: string): boolean; + +// @public (undocumented) +export function isValidAPIKey(apiKey: string): boolean; + +// @public (undocumented) +export const isValidURL: (url: string) => boolean | undefined; + +// @public (undocumented) +export const JWT: { + readonly ALGORITHM_RS256: "RS256"; + readonly GRANT_TYPE_JWT_BEARER: "urn:ietf:params:oauth:grant-type:jwt-bearer"; + readonly ISSUER_SDK: "sdk"; + readonly SIGNED_TOKEN_PREFIX: "signed_token_"; + readonly ROLE_PREFIX: "role:"; +}; + +// @public (undocumented) +export enum LogLevel { + // (undocumented) + DEBUG = "DEBUG", + // (undocumented) + ERROR = "ERROR", + // (undocumented) + INFO = "INFO", + // (undocumented) + OFF = "OFF", + // (undocumented) + WARN = "WARN" +} + +// @public (undocumented) +export const LogLevelOptions: { + DEBUG: { + showDebugLogs: boolean; + showInfoLogs: boolean; + showWarnLogs: boolean; + showErrorLogs: boolean; + }; + INFO: { + showDebugLogs: boolean; + showInfoLogs: boolean; + showWarnLogs: boolean; + showErrorLogs: boolean; + }; + WARN: { + showDebugLogs: boolean; + showInfoLogs: boolean; + showWarnLogs: boolean; + showErrorLogs: boolean; + }; + ERROR: { + showDebugLogs: boolean; + showInfoLogs: boolean; + showWarnLogs: boolean; + showErrorLogs: boolean; + }; + OFF: { + showDebugLogs: boolean; + showInfoLogs: boolean; + showWarnLogs: boolean; + showErrorLogs: boolean; + }; +}; + +// @public (undocumented) +export const logs: { + infoLogs: { + EMPTY_BEARER_TOKEN: string; + BEARER_TOKEN_EXPIRED: string; + GENERATE_BEARER_TOKEN_TRIGGERED: string; + GENERATE_BEARER_TOKEN_SUCCESS: string; + GENERATE_SIGNED_DATA_TOKEN_SUCCESS: string; + INITIALIZE_CLIENT: string; + VALIDATING_VAULT_CONFIG: string; + VALIDATING_CONNECTION_CONFIG: string; + VAULT_CONTROLLER_INITIALIZED: string; + DETECT_CONTROLLER_INITIALIZED: string; + CONNECTION_CONTROLLER_INITIALIZED: string; + vaultId_CONFIG_EXISTS: string; + vaultId_CONFIG_DOES_NOT_EXIST: string; + connectionId_CONFIG_EXISTS: string; + connectionId_CONFIG_DOES_NOT_EXIST: string; + CURRENT_LOG_LEVEL: string; + CLIENT_INITIALIZED: string; + VALIDATE_INSERT_INPUT: string; + VALIDATE_DETOKENIZE_INPUT: string; + VALIDATE_TOKENIZE_INPUT: string; + VALIDATE_FILE_UPLOAD_INPUT: string; + VALIDATE_GET_INPUT: string; + VALIDATE_QUERY_INPUT: string; + VALIDATE_DELETE_INPUT: string; + VALIDATE_UPDATE_INPUT: string; + VALIDATE_CONNECTION_CONFIG: string; + INSERT_DATA_SUCCESS: string; + FILE_UPLOAD_DATA_SUCCESS: string; + DETOKENIZE_SUCCESS: string; + GET_SUCCESS: string; + UPDATE_SUCCESS: string; + DELETE_SUCCESS: string; + TOKENIZE_SUCCESS: string; + QUERY_SUCCESS: string; + BEARER_TOKEN_LISTENER: string; + BEARER_TOKEN_RESOLVED: string; + REUSE_BEARER_TOKEN: string; + REUSE_KEY: string; + CONTROLLER_INITIALIZED: string; + INSERT_TRIGGERED: string; + DETOKENIZE_TRIGGERED: string; + TOKENIZE_TRIGGERED: string; + GET_BY_ID_TRIGGERED: string; + GET_TRIGGERED: string; + QUERY_TRIGGERED: string; + UPLOAD_FILE_TRIGGERED: string; + INVOKE_CONNECTION_TRIGGERED: string; + DELETE_TRIGGERED: string; + DELETE_REQUEST_RESOLVED: string; + TOKENIZE_REQUEST_RESOLVED: string; + QUERY_REQUEST_RESOLVED: string; + FILE_UPLOAD_REQUEST_RESOLVED: string; + EMIT_REQUEST: string; + DETOKENIZE_REQUEST_RESOLVED: string; + INSERT_REQUEST_RESOLVED: string; + INSERT_BATCH_REQUEST_RESOLVED: string; + GET_REQUEST_RESOLVED: string; + INVOKE_CONNECTION_REQUEST_RESOLVED: string; + GENERATE_SIGNED_DATA_TOKENS_TRIGGERED: string; + UPDATE_TRIGGERED: string; + UPDATE_REQUEST_RESOLVED: string; + UNABLE_TO_GENERATE_SDK_METRIC: string; + USING_BEARER_TOKEN: string; + USING_KEY: string; + USING_CREDENTIALS_STRING: string; + USING_PATH: string; + USING_SKYFLOW_CREDENTIALS_ENV: string; + DEIDENTIFY_STRING_TRIGGERED: string; + VALIDATE_DEIDENTIFY_STRING_REQUEST: string; + DEIDENTIFY_STRING_REQUEST_RESOLVED: string; + DEIDENTIFY_STRING_SUCCESS: string; + REIDENTIFY_STRING_TRIGGERED: string; + VALIDATE_REIDENTIFY_STRING_REQUEST: string; + REIDENTIFY_STRING_REQUEST_RESOLVED: string; + REIDENTIFY_STRING_SUCCESS: string; + DEIDENTIFY_FILE_TRIGGERED: string; + VALIDATE_DEIDENTIFY_FILE_REQUEST: string; + DEIDENTIFY_FILE_REQUEST_RESOLVED: string; + DEIDENTIFY_FILE_RUN_SUBMITTED: string; + DEIDENTIFY_FILE_SUCCESS: string; + REIDENTIFY_FILE_TRIGGERED: string; + VALIDATE_REIDENTIFY_FILE_REQUEST: string; + REIDENTIFY_FILE_REQUEST_RESOLVED: string; + REIDENTIFY_FILE_SUCCESS: string; + GET_RUN_TRIGGERED: string; + VALIDATE_GET_RUN_REQUEST: string; + GET_RUN_REQUEST_RESOLVED: string; + GET_RUN_SUCCESS: string; + CHECK_GUARDRAILS_TRIGGERED: string; + VALIDATE_CHECK_GUARDRAILS_REQUEST: string; + CHECK_GUARDRAILS_REQUEST_RESOLVED: string; + CHECK_GUARDRAILS_SUCCESS: string; + POLL_RUN_STARTED: string; + POLL_RUN_TERMINAL: string; + POLL_RUN_BUDGET_EXHAUSTED: string; + OUTPUT_FILE_WRITTEN: string; + OUTPUT_NOT_WRITTEN_FOR_OUTPUT_TYPE: string; + }; + errorLogs: { + VAULT_CONFIG_KEY_MISSING: string; + EMPTY_VAULT_ID: string; + INVALID_VAULT_ID: string; + EMPTY_CLUSTER_ID: string; + INVALID_CLUSTER_ID: string; + INVALID_ENV: string; + EMPTY_CONNECTION_ID: string; + INVALID_CONNECTION_ID: string; + EMPTY_CONNECTION_URL: string; + INVALID_CONNECTION_URL: string; + INVALID_CONNECTION_URL_TYPE: string; + EMPTY_CREDENTIALS_PATH: string; + EMPTY_CREDENTIALS_STRING: string; + EMPTY_TOKEN_VALUE: string; + EMPTY_API_KEY_VALUE: string; + INVALID_KEY: string; + CLIENT_ID_NOT_FOUND: string; + TOKEN_URI_NOT_FOUND: string; + KEY_ID_NOT_FOUND: string; + PRIVATE_KEY_NOT_FOUND: string; + NOT_A_VALID_JSON: string; + FILE_NOT_FOUND: string; + EMPTY_FILE: string; + CREDENTIALS_CONTENT_EMPTY: string; + EXPECTED_STRING_PARAMETER: string; + SCOPED_ROLES_EMPTY: string; + EXPECTED_ROLE_ID_PARAMETER: string; + DATA_TOKENS_NOT_FOUND: string; + DATA_TOKENS_EMPTY: string; + EXPECTED_DATA_TOKENS_PARAMETER: string; + EXPECTED_TIME_TO_LIVE_PARAMETER: string; + DETOKENIZE_REQUEST_REJECTED: string; + TOKENIZE_REQUEST_REJECTED: string; + INVOKE_CONNECTION_REQUEST_REJECTED: string; + QUERY_REQUEST_REJECTED: string; + INSERT_REQUEST_REJECTED: string; + INSERT_BATCH_REQUEST_REJECTED: string; + FILE_UPLOAD_REQUEST_REJECTED: string; + GET_REQUEST_REJECTED: string; + SEND_INVOKE_CONNECTION_REJECTED: string; + UPDATE_REQUEST_REJECTED: string; + DELETE_REQUEST_REJECTED: string; + ERROR_OCCURED: string; + EMPTY_QUERY: string; + QUERY_IS_REQUIRED: string; + VALUES_IS_REQUIRED_TOKENIZE: string; + EMPTY_VALUES_IN_TOKENIZE: string; + EMPTY_COLUMN_GROUP_IN_COLUMN_VALUES: string; + EMPTY_TABLE_IN_DELETE: string; + INVALID_TABLE_IN_DELETE: string; + EMPTY_IDS_IN_DELETE: string; + INVALID_ID_IN_DELETE: string; + EMPTY_TABLE_IN_INSERT: string; + INVALID_TABLE_IN_INSERT: string; + EMPTY_TABLE_IN_GET: string; + INVALID_TABLE_IN_GET: string; + EMPTY_IDS_IN_GET: string; + INVALID_ID_IN_GET: string; + EMPTY_TOKENS_IN_DETOKENIZE: string; + INVALID_TOKENS_IN_DETOKENIZE: string; + INVALID_TOKEN_IN_DETOKENIZE: string; + EMPTY_TABLE_IN_GET_COLUMN: string; + INVALID_TABLE_IN_GET_COLUMN: string; + EMPTY_COLUMN_NAME_IN_GET_COLUMN: string; + INVALID_COLUMN_NAME_IN_GET_COLUMN: string; + EMPTY_COLUMN_VALUES_IN_GET_COLUMN: string; + INVALID_COLUMN_VALUES_IN_GET_COLUMN: string; + INVALID_COLUMN_VALUE_IN_GET_COLUMN: string; + EMPTY_SKYFLOW_ID_IN_UPDATE: string; + INVALID_SKYFLOW_ID_IN_UPDATE: string; + EMPTY_TABLE_IN_UPDATE: string; + INVALID_TABLE_IN_UPDATE: string; + EMPTY_TABLE_IN_FILE_UPLOAD: string; + INVALID_TABLE_IN_FILE_UPLOAD: string; + EMPTY_SKYFLOW_ID_IN_FILE_UPLOAD: string; + INVALID_SKYFLOW_ID_IN_FILE_UPLOAD: string; + INVALID_TOKEN_URI: string; + DEIDENTIFY_STRING_REQUEST_NULL: string; + INVALID_TEXT_IN_DEIDENTIFY_STRING: string; + DEIDENTIFY_STRING_REQUEST_REJECTED: string; + REIDENTIFY_STRING_REQUEST_NULL: string; + INVALID_TEXT_IN_REIDENTIFY_STRING: string; + INVALID_REDACTION_LEVEL: string; + REIDENTIFY_STRING_REQUEST_REJECTED: string; + DEIDENTIFY_FILE_REQUEST_NULL: string; + INVALID_DATA_SOURCE_IN_DEIDENTIFY_FILE: string; + INVALID_VALUE_IN_DEIDENTIFY_FILE: string; + MULTIPLE_FILE_SOURCES_IN_DEIDENTIFY_FILE: string; + MISSING_FILE_SOURCE_IN_DEIDENTIFY_FILE: string; + INVALID_FILE_IN_DEIDENTIFY_FILE: string; + FILE_NOT_FOUND_IN_DEIDENTIFY_FILE: string; + FAILED_TO_ENCODE_FILE_IN_DEIDENTIFY_FILE: string; + INVALID_WAIT_TIME_IN_POLL_OPTIONS: string; + INVALID_MAX_ATTEMPTS_IN_POLL_OPTIONS: string; + OUTPUT_DIRECTORY_WITHOUT_POLL_OPTIONS: string; + DEIDENTIFY_FILE_REQUEST_REJECTED: string; + OUTPUT_DIRECTORY_NOT_CREATED: string; + OUTPUT_FILE_NOT_WRITTEN: string; + REIDENTIFY_FILE_REQUEST_NULL: string; + INVALID_DATA_SOURCE_IN_REIDENTIFY_FILE: string; + INVALID_VALUE_IN_REIDENTIFY_FILE: string; + REIDENTIFY_FILE_REQUEST_REJECTED: string; + GET_RUN_REQUEST_NULL: string; + INVALID_RUN_ID_IN_GET_RUN: string; + GET_RUN_REQUEST_REJECTED: string; + CHECK_GUARDRAILS_REQUEST_NULL: string; + INVALID_TEXT_IN_CHECK_GUARDRAILS: string; + INVALID_DENY_TOPICS_IN_CHECK_GUARDRAILS: string; + CHECK_GUARDRAILS_REQUEST_REJECTED: string; + }; + warnLogs: { + DEPRECATED_FILE_UPLOAD_CONSTRUCTOR: string; + DEPRECATED_FILE_UPLOAD_SKYFLOW_ID: string; + DEPRECATED_SET_DOWNLOAD_URL: string; + DEPRECATED_GET_DOWNLOAD_URL: string; + DEPRECATED_SKYFLOW_ID_PROPERTY: string; + DEPRECATED_REQUEST_ID_PROPERTY: string; + DEPRECATED_ROLE_IDS_PROPERTY: string; + DEPRECATED_HTTP_CODE_PROPERTY: string; + DEPRECATED_HTTP_STATUS_PROPERTY: string; + DEPRECATED_GRPC_CODE_PROPERTY: string; + }; +}; + // @public -export class GetTokensOptions extends FlowVaultOptions { - constructor(); +export enum MaskingMethod { + // (undocumented) + BLACKBOX = "BLACKBOX", + // (undocumented) + BLUR = "BLUR", + // (undocumented) + NONE = "NONE" +} + +// @public +export interface Media { + // (undocumented) + audio?: Audio_2; + // (undocumented) + document?: Document_2; + // (undocumented) + image?: Image_2; +} + +// @public (undocumented) +export enum MessageType { + // (undocumented) + ERROR = "ERROR", + // (undocumented) + LOG = "LOG", + // (undocumented) + WARN = "WARN" } // @public -export interface GetTokensRecord { +export interface Metrics { // (undocumented) - tokenGroupName: string; + characterCount: number | null; + duration: number | null; + pages: number | null; + size: number | null; + slides: number | null; // (undocumented) - value: unknown; + wordCount: number | null; } // @public -export class GetTokensRequest extends BaseGetTokensRequest { - constructor(records: GetTokensRecord[]); +export enum ObjectDeidentificationType { // (undocumented) - get records(): GetTokensRecord[]; - set records(value: GetTokensRecord[]); + REDACT = "REDACT", + // (undocumented) + UNKNOWN = "UNKNOWN", + // (undocumented) + UNREDACT = "UNREDACT" } // @public -export class GetTokensResponse extends BaseGetTokensResponse { - constructor(input: { - records: GetTokensResponseRecord[]; - }); +export interface ObjectEntity { // (undocumented) - readonly records: GetTokensResponseRecord[]; + deidentificationType?: ObjectDeidentificationType; + // (undocumented) + entityType?: ObjectEntityType; } -// @public (undocumented) -export interface GetTokensResponseRecord { +// @public +export enum ObjectEntityType { // (undocumented) - error?: string | null; + ALL = "ALL", // (undocumented) - httpCode: number; + FACE = "FACE", // (undocumented) - requestId?: string | null; + LICENSE_PLATE = "LICENSE_PLATE", // (undocumented) - token?: string | null; + LOGO = "LOGO", // (undocumented) - tokenGroupName: string | null; + NONE = "NONE", // (undocumented) - value: unknown; + SIGNATURE = "SIGNATURE" } -// @public -export class InsertOptions extends FlowVaultOptions { - constructor(); +// @public (undocumented) +export function objectToXML(obj: any, rootName?: string): string; + +// @public (undocumented) +export enum OrderByEnum { // (undocumented) - getUpsert(): UpsertOptions | undefined; + ASCENDING = "ASCENDING", // (undocumented) - setUpsert(upsert: UpsertOptions): void; + DESCENDING = "DESCENDING", + // (undocumented) + NONE = "NONE" } // @public -export interface InsertRecord { +export enum OutputTranscriptionType { // (undocumented) - data: Record; + DIARIZED_TRANSCRIPTION = "DIARIZED_TRANSCRIPTION", // (undocumented) - table?: string; + MEDICAL_DIARIZED_TRANSCRIPTION = "MEDICAL_DIARIZED_TRANSCRIPTION", // (undocumented) - tokens?: Record; - upsert?: UpsertOptions; -} - -// @public -export class InsertRequest extends BaseInsertRequest { - constructor(tableName: string, records: InsertRecord[]); + MEDICAL_TRANSCRIPTION = "MEDICAL_TRANSCRIPTION", // (undocumented) - get records(): InsertRecord[]; - set records(value: InsertRecord[]); + NONE = "NONE", // (undocumented) - get tableName(): string; - set tableName(value: string); + TRANSCRIPTION = "TRANSCRIPTION" } -// @public -export class InsertResponse extends BaseInsertResponse { - constructor(input: { - records: InsertResponseRecord[]; - }); - // (undocumented) - readonly records: InsertResponseRecord[]; -} +// @public (undocumented) +export const parameterizedString: (message: string, ...args: Array) => string; // @public (undocumented) -export interface InsertResponseRecord { +export interface PathCredentials { // (undocumented) - data?: Record | null; + context?: string | Record; // (undocumented) - error?: string | null; + path: string; // (undocumented) - hashedData?: Record | null; + roles?: Array; // (undocumented) - httpCode: number; + tokenUri?: string; +} + +// @public +export interface Pdf { // (undocumented) - requestId?: string | null; + density?: number; // (undocumented) - skyflowId?: string | null; + maxResolution?: number; // (undocumented) - tableName?: string | null; + processingMode?: PdfProcessingMode; +} + +// @public +export enum PdfProcessingMode { // (undocumented) - tokens?: Record | null; + NONE = "NONE", + // (undocumented) + OCR = "OCR", + // (undocumented) + TEXT_LAYER = "TEXT_LAYER" +} + +// @public +export const POLL_OPTIONS: { + readonly DEFAULT_WAIT_TIME_SECONDS: 60; + readonly MAX_WAIT_TIME_SECONDS: 300; + readonly DEFAULT_MAX_ATTEMPTS: 23; + readonly MAX_MAX_ATTEMPTS: 23; + readonly MAX_DELAY_SECONDS: 32; +}; + +// @public +export interface PollOptions { + maxAttempts?: number; + waitTime?: number; } +// @public (undocumented) +export const printLog: (message: string, messageType: MessageType, logLevel?: LogLevel) => void; + // @public export class QueryOptions extends FlowVaultOptions { constructor(); @@ -438,6 +2012,12 @@ export interface QueryResponseMetadata { columns: string[]; } +// @public (undocumented) +export interface QueryResponseType { + // (undocumented) + [key: string]: unknown; +} + // @public export interface RedactionLevel extends RedactionRule { // (undocumented) @@ -448,6 +2028,91 @@ export interface RedactionLevel extends RedactionRule { tokenGroupName?: string; } +// @public +export interface RedactionRule { + entityName?: EntityType; + redactionPattern?: string; + redactionType?: RedactionType; + tokenGroupName?: string; +} + +// @public (undocumented) +export enum RedactionType { + // (undocumented) + DEFAULT = "DEFAULT", + // (undocumented) + MASKED = "MASKED", + // (undocumented) + PLAIN_TEXT = "PLAIN_TEXT", + // (undocumented) + REDACTED = "REDACTED" +} + +// @public +export enum ReidentifyFileDataFormat { + // (undocumented) + CSV = "CSV", + // (undocumented) + DOCX = "DOCX", + // (undocumented) + JSON = "JSON", + // (undocumented) + JSONL = "JSONL", + // (undocumented) + PPTX = "PPTX", + // (undocumented) + TXT = "TXT", + // (undocumented) + XLSX = "XLSX", + // (undocumented) + XML = "XML" +} + +// @public +export interface ReidentifyFileOutput { + processedFile: string | null; + processedFileExtension: ReidentifyFileDataFormat | null; + processedFileType: string | null; +} + +// @public +export class ReidentifyFileRequest { + constructor(dataSource: ReidentifyFileSource, value: string, fields?: ReidentifyFileRequestFields); + // (undocumented) + get dataFormat(): ReidentifyFileDataFormat | undefined; + set dataFormat(value: ReidentifyFileDataFormat | undefined); + get dataSource(): ReidentifyFileSource; + set dataSource(value: ReidentifyFileSource); + // (undocumented) + get redactionLevel(): RedactionRule[] | undefined; + set redactionLevel(value: RedactionRule[] | undefined); + get value(): string; + set value(value: string); +} + +// @public +export interface ReidentifyFileRequestFields { + dataFormat?: ReidentifyFileDataFormat; + redactionLevel?: RedactionRule[]; +} + +// @public +export class ReidentifyFileResponse { + constructor(input: { + status: DetectRunStatus; + outputType: ReidentifyFileSource | null; + output: ReidentifyFileOutput[]; + metrics: Metrics | null; + }); + readonly metrics: Metrics | null; + readonly output: ReidentifyFileOutput[]; + readonly outputType: ReidentifyFileSource | null; + readonly status: DetectRunStatus; +} + +// @public +export type ReidentifyFileSource = string; + // @public export enum ReidentifySourceType { // (undocumented) @@ -456,6 +2121,39 @@ export enum ReidentifySourceType { PRESIGNED_URL = "PRESIGNED_URL" } +// @public +export class ReidentifyStringRequest { + constructor(text: string, fields?: ReidentifyStringRequestFields); + // (undocumented) + get redactionLevel(): RedactionRule[] | undefined; + set redactionLevel(value: RedactionRule[] | undefined); + get text(): string; + set text(value: string); +} + +// @public +export interface ReidentifyStringRequestFields { + redactionLevel?: RedactionRule[]; +} + +// @public +export class ReidentifyStringResponse { + constructor(input: { + processedText: string | null; + metrics: Metrics | null; + }); + readonly metrics: Metrics | null; + readonly processedText: string | null; +} + +// @public (undocumented) +export function removeSDKVersion(message: string): string; + +// @public (undocumented) +export const REQUEST: { + readonly ID_KEY: "x-request-id"; +}; + // @public export class RequestContext { constructor(operation: string, batchIndex?: number, totalBatches?: number); @@ -470,6 +2168,89 @@ export class RequestContext { // @public export type RequestInterceptor = (context: RequestContext) => void; +// @public (undocumented) +export enum RequestMethod { + // (undocumented) + GET = "GET", + // (undocumented) + PATCH = "PATCH", + // (undocumented) + POST = "POST", + // (undocumented) + PUT = "PUT" +} + +// @public +export enum ReturnEntitiesType { + // (undocumented) + ALL = "ALL", + // (undocumented) + EXCLUDE_SENSITIVE_DATA = "EXCLUDE_SENSITIVE_DATA", + // (undocumented) + NONE = "NONE" +} + +// @public (undocumented) +export const SDK: { + readonly METRICS_HEADER_KEY: "sky-metadata"; +}; + +// @public (undocumented) +export interface SdkIdentity { + // (undocumented) + name: string; + // (undocumented) + version: string; +} + +// @public (undocumented) +export type ServiceAccountResponseError = { + rawResponse?: { + headers?: { + get: (header: string) => string | undefined; + }; + }; + body: any; + response?: { + status?: number | string; + }; + message: string; + [key: string]: any; +}; + +// @public (undocumented) +export function setSdkIdentity(id: SdkIdentity): void; + +// @public +export interface ShiftDates { + // (undocumented) + maxDays?: number; + // (undocumented) + minDays?: number; +} + +// @public (undocumented) +export type SignedDataTokensOptions = { + dataTokens: string[]; + timeToLive?: number; + ctx?: string | Record; + logLevel?: LogLevel; + tokenUri?: string; +}; + +// @public (undocumented) +export type SignedDataTokensResponse = { + token: string; + signedToken: string; +}; + +// @public (undocumented) +export const SKYFLOW: { + readonly ID: "skyflowId"; + readonly LEGACY_ID: "skyflow_id"; + readonly AUTH_HEADER_KEY: "x-skyflow-authorization"; +}; + // Warning: (ae-forgotten-export) The symbol "ClientEntry" needs to be exported by the entry point index.d.ts // // @public @@ -489,6 +2270,64 @@ class Skyflow extends BaseSkyflow { export { Skyflow } export default Skyflow; +// Warning: (ae-forgotten-export) The symbol "_SKYFLOW_ERROR_CODE" needs to be exported by the entry point index.d.ts +// +// @public (undocumented) +export const SKYFLOW_ERROR_CODE: typeof _SKYFLOW_ERROR_CODE; + +// @public (undocumented) +export interface SkyflowConfig { + // (undocumented) + connectionConfigs?: ConnectionConfig[]; + // (undocumented) + logLevel?: LogLevel; + // (undocumented) + skyflowCredentials?: Credentials; + // (undocumented) + vaultConfigs?: VaultConfig[]; +} + +// @public (undocumented) +export class SkyflowError extends Error { + constructor(errorCode: ISkyflowError, args?: Array); + // (undocumented) + error?: ISkyflowError; +} + +// @public (undocumented) +export interface SkyflowRecordError { + // (undocumented) + error: string; + // (undocumented) + httpCode?: string | number | null; + // @deprecated (undocumented) + request_ID?: string | null; + // (undocumented) + requestId: string | null; + // (undocumented) + requestIndex?: number | null; + // (undocumented) + token?: string | null; +} + +// @public (undocumented) +export interface StringCredentials { + // (undocumented) + context?: string | Record; + // (undocumented) + credentialsString: string; + // (undocumented) + roles?: Array; + // (undocumented) + tokenUri?: string; +} + +// @public (undocumented) +export interface StringKeyValueMapType { + // (undocumented) + [key: string]: object | string; +} + // @public export interface Token { // (undocumented) @@ -499,6 +2338,12 @@ export interface Token { tokenGroupName?: string; } +// @public (undocumented) +export interface TokenCredentials { + // (undocumented) + token: string; +} + // @public export interface TokenGroupRedaction { // (undocumented) @@ -507,6 +2352,51 @@ export interface TokenGroupRedaction { tokenGroupName: string; } +// @public (undocumented) +export interface TokenizeRequestType { + // (undocumented) + columnGroup: string; + // (undocumented) + value: string; +} + +// @public (undocumented) +export enum TokenMode { + // (undocumented) + DISABLE = "DISABLE", + // (undocumented) + ENABLE = "ENABLE", + // (undocumented) + ENABLE_STRICT = "ENABLE_STRICT" +} + +// @public (undocumented) +export type TokenResponse = { + accessToken: string; + tokenType: string; +}; + +// @public +export interface Transformation { + // (undocumented) + shiftDates?: ShiftDates; +} + +// @public (undocumented) +export const TYPES: { + INSERT: string; + INSERT_BATCH: string; + DETOKENIZE: string; + TOKENIZE: string; + DELETE: string; + UPDATE: string; + GET: string; + FILE_UPLOAD: string; + QUERY: string; + DETECT: string; + INVOKE_CONNECTION: string; +}; + // @public export class UpdateOptions extends FlowVaultOptions { constructor(); @@ -652,8 +2542,49 @@ export interface UpsertOptions { updateType: string; } +// @public (undocumented) +export const URL_PROTOCOL: { + readonly HTTPS: "https"; +}; + +// @public (undocumented) +export const validateConnectionConfig: (connectionConfig: ConnectionConfig, logLevel?: LogLevel) => void; + +// @public (undocumented) +export const validateCredentialsWithId: (credentials: Credentials, type: string, typeId: string, id: string, logLevel?: LogLevel) => void; + +// @public (undocumented) +export const validateSkyflowConfig: (config: SkyflowConfig, logLevel?: LogLevel) => void; + +// @public (undocumented) +export const validateSkyflowCredentials: (credentials: Credentials, logLevel?: LogLevel) => void; + +// @public (undocumented) +export function validateToken(token: string): string; + +// @public (undocumented) +export const validateUpdateConnectionConfig: (connectionConfig: ConnectionConfig, logLevel?: LogLevel) => void; + +// @public (undocumented) +export const validateUpdateVaultConfig: (vaultConfig: VaultConfig, logLevel?: LogLevel) => void; + +// @public (undocumented) +export const validateVaultConfig: (vaultConfig: VaultConfig, logLevel?: LogLevel) => void; -export * from "skyflow-common"; +// @public (undocumented) +export interface VaultConfig { + // (undocumented) + clusterId: string; + // (undocumented) + credentials?: Credentials; + // (undocumented) + env?: Env; + maxRetries?: number; + timeout?: number; + // (undocumented) + vaultId: string; + vaultUrl?: string; +} // (No @packageDocumentation comment for this package) diff --git a/flowvault/samples/detect-api/deidentify-file.ts b/flowvault/samples/detect-api/deidentify-file.ts index 8c010e30..0cc358fd 100644 --- a/flowvault/samples/detect-api/deidentify-file.ts +++ b/flowvault/samples/detect-api/deidentify-file.ts @@ -9,7 +9,7 @@ import { DeidentifyFileRequest, DeidentifyFileResponse, DeidentifySourceType, - FileDataFormat, + DeidentifyFileDataFormat, } from 'skyflow-flowvault-node'; /** @@ -64,7 +64,7 @@ async function performDeidentifyFile() { const remoteFileRequest = new DeidentifyFileRequest({ dataSource: DeidentifySourceType.PRESIGNED_URL, value: '', - dataFormat: FileDataFormat.PDF, + dataFormat: DeidentifyFileDataFormat.PDF, configurationId: '', }); const remoteSubmitted = await skyflowClient diff --git a/flowvault/samples/detect-api/get-run.ts b/flowvault/samples/detect-api/get-run.ts index 9357731b..c1ea1e1a 100644 --- a/flowvault/samples/detect-api/get-run.ts +++ b/flowvault/samples/detect-api/get-run.ts @@ -55,7 +55,7 @@ async function performGetRun() { for (const output of run.output) { // output.processedFile is base64 or a presigned URL, per run.outputType // output.processedFileType is e.g. REDACTED_FILE or ENTITIES - // output.processedFileExtension is a FileDataFormat such as PDF + // output.processedFileExtension is a DeidentifyFileDataFormat such as PDF console.log(`${output.processedFileType} (${output.processedFileExtension})`); } console.log('Output type:', run.outputType); diff --git a/flowvault/samples/detect-api/reidentify-file.ts b/flowvault/samples/detect-api/reidentify-file.ts index 9b518506..6fde7fb5 100644 --- a/flowvault/samples/detect-api/reidentify-file.ts +++ b/flowvault/samples/detect-api/reidentify-file.ts @@ -10,7 +10,7 @@ import { ReidentifyFileRequest, ReidentifyFileResponse, ReidentifySourceType, - FileDataFormat, + ReidentifyFileDataFormat, DetectRunStatus, } from 'skyflow-flowvault-node'; @@ -50,7 +50,7 @@ async function performReidentifyFile() { // formats that support VAULT_TOKEN: TXT, CSV, JSON, JSONL, XML, DOCX, XLSX and PPTX. const content = fs.readFileSync('').toString('base64'); const request = new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { - dataFormat: FileDataFormat.TXT, + dataFormat: ReidentifyFileDataFormat.TXT, redactionLevel: [ { tokenGroupName: '', redactionPattern: '' }, ], diff --git a/skyvault/README.md b/skyvault/README.md index e9fe2381..5b7fcf29 100644 --- a/skyvault/README.md +++ b/skyvault/README.md @@ -809,7 +809,7 @@ File de-identification is asynchronous on the API: the submit call returns a run The file is supplied in one of three ways: a `dataSource` plus `value` (`BASE64` content, a `SKYFLOW_ID`, or a `PRESIGNED_URL`), a `file` (a web `File`, Node.js 20+), or a `filePath`. A local file is read, base64-encoded, and its `dataFormat` inferred from the extension. Supply exactly one of `value`, `file` or `filePath`. `configurationId` and `configuration` follow the same rules as for strings. -`dataFormat` accepts every `FileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. +`dataFormat` accepts every `DeidentifyFileDataFormat`: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML` and `ZIP`. The `VAULT_TOKEN` de-identification type is supported only for `TXT`, `CSV`, `JSON`, `JSONL`, `XML`, `DOCX`, `XLSX` and `PPTX`. `outputDirectory` is meaningful only when polling. Setting it without `pollOptions` fails client-side validation before any request is sent. @@ -951,12 +951,12 @@ console.log(rendered.processedText); ```typescript import * as fs from 'fs'; -import { ReidentifyFileRequest, ReidentifySourceType, FileDataFormat, DetectRunStatus } from 'skyflow-node'; +import { ReidentifyFileRequest, ReidentifySourceType, ReidentifyFileDataFormat, DetectRunStatus } from 'skyflow-node'; const content = fs.readFileSync('tokenized.txt').toString('base64'); const response = await skyflowClient.detect().reidentifyFile( - new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: FileDataFormat.TXT }), + new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { dataFormat: ReidentifyFileDataFormat.TXT }), ); if (response.status === DetectRunStatus.SUCCESS) { @@ -1001,7 +1001,7 @@ switch (run.status) { case DetectRunStatus.SUCCESS: for (const output of run.output) { // output.processedFileType is e.g. REDACTED_FILE or ENTITIES - // output.processedFileExtension is a FileDataFormat such as PDF + // output.processedFileExtension is a DeidentifyFileDataFormat such as PDF if (run.outputType === DeidentifySourceType.BASE64) { const bytes = Buffer.from(output.processedFile ?? '', 'base64'); fs.writeFileSync(`processed-${output.processedFileType}.${output.processedFileExtension?.toLowerCase()}`, bytes); diff --git a/skyvault/api-report/skyflow-node.api.md b/skyvault/api-report/skyflow-node.api.md index e068df04..f4df90d3 100644 --- a/skyvault/api-report/skyflow-node.api.md +++ b/skyvault/api-report/skyflow-node.api.md @@ -4,139 +4,232 @@ ```ts -import { ApiKeyCredentials } from 'skyflow-common'; -import { Audio as Audio_2 } from 'skyflow-common'; -import { AuthInfo } from 'skyflow-common'; -import { BaseDeleteRequest } from 'skyflow-common'; -import { BaseDeleteResponse } from 'skyflow-common'; -import { BaseDetectController } from 'skyflow-common'; -import { BaseDetokenizeRequest } from 'skyflow-common'; -import { BaseDetokenizeResponse } from 'skyflow-common'; -import { BaseFileUploadRequest } from 'skyflow-common'; -import { BaseFileUploadResponse } from 'skyflow-common'; -import { BaseGetRequest } from 'skyflow-common'; -import { BaseGetResponse } from 'skyflow-common'; -import { BaseInsertRequest } from 'skyflow-common'; -import { BaseInsertResponse } from 'skyflow-common'; -import { BaseInvokeConnectionRequest } from 'skyflow-common'; -import { BaseInvokeConnectionResponse } from 'skyflow-common'; -import { BaseOptions } from 'skyflow-common'; -import { BaseQueryRequest } from 'skyflow-common'; -import { BaseQueryResponse } from 'skyflow-common'; -import { BaseSkyflow } from 'skyflow-common'; -import { BaseTokenizeRequest } from 'skyflow-common'; -import { BaseTokenizeResponse } from 'skyflow-common'; -import { BaseUpdateRequest } from 'skyflow-common'; -import { BaseUpdateResponse } from 'skyflow-common'; -import { BaseVaultEntry } from 'skyflow-common'; -import { BearerTokenOptions } from 'skyflow-common'; -import { Bleep } from 'skyflow-common'; import { Blob as Blob_2 } from 'buffer'; -import { CheckGuardrailsRequest } from 'skyflow-common'; -import { CheckGuardrailsRequestFields } from 'skyflow-common'; -import { CheckGuardrailsResponse } from 'skyflow-common'; -import { ConnectionConfig } from 'skyflow-common'; -import { Credentials } from 'skyflow-common'; -import { CustomType } from 'skyflow-common'; -import { DeidentificationType } from 'skyflow-common'; -import { DeidentifyFileRequest } from 'skyflow-common'; -import { DeidentifyFileRequestFields } from 'skyflow-common'; -import { DeidentifyFileResponse } from 'skyflow-common'; -import { DeidentifyStringRequest } from 'skyflow-common'; -import { DeidentifyStringRequestFields } from 'skyflow-common'; -import { DeidentifyStringResponse } from 'skyflow-common'; -import { Detect } from 'skyflow-common'; -import { DetectConfiguration } from 'skyflow-common'; -import { DetectedEntity } from 'skyflow-common'; -import { DetectRunFields } from 'skyflow-common'; -import { DetectRunStatus } from 'skyflow-common'; -import { Document as Document_2 } from 'skyflow-common'; -import { Entity } from 'skyflow-common'; -import { EntityLocation } from 'skyflow-common'; -import { EntityType } from 'skyflow-common'; -import { Env } from 'skyflow-common'; -import { FileDataFormat } from 'skyflow-common'; -import { FileMapping } from 'skyflow-common'; -import { FileOutput } from 'skyflow-common'; import * as fs from 'fs'; -import { generateBearerToken } from 'skyflow-common'; -import { generateBearerTokenFromCreds } from 'skyflow-common'; -import { generateSignedDataTokens } from 'skyflow-common'; -import { generateSignedDataTokensFromCreds } from 'skyflow-common'; -import { GenerateTokenOptions } from 'skyflow-common'; -import { GetResponseData } from 'skyflow-common'; -import { GetRunRequest } from 'skyflow-common'; -import { GetRunResponse } from 'skyflow-common'; -import { GuardrailsValidation } from 'skyflow-common'; -import { Image as Image_2 } from 'skyflow-common'; -import { IncludePrefix } from 'skyflow-common'; -import { InsertResponseType } from 'skyflow-common'; -import { isExpired } from 'skyflow-common'; -import { LogLevel } from 'skyflow-common'; -import { MaskingMethod } from 'skyflow-common'; -import { Media } from 'skyflow-common'; -import { Metrics } from 'skyflow-common'; -import { ObjectDeidentificationType } from 'skyflow-common'; -import { ObjectEntity } from 'skyflow-common'; -import { ObjectEntityType } from 'skyflow-common'; -import { OrderByEnum } from 'skyflow-common'; -import { OutputTranscriptionType } from 'skyflow-common'; -import { PathCredentials } from 'skyflow-common'; -import { Pdf } from 'skyflow-common'; -import { PdfProcessingMode } from 'skyflow-common'; -import { POLL_OPTIONS } from 'skyflow-common'; -import { PollOptions } from 'skyflow-common'; -import { QueryResponseType } from 'skyflow-common'; -import { RedactionRule } from 'skyflow-common'; -import { RedactionType } from 'skyflow-common'; -import { ReidentifyFileRequest } from 'skyflow-common'; -import { ReidentifyFileRequestFields } from 'skyflow-common'; -import { ReidentifyFileResponse } from 'skyflow-common'; -import { ReidentifyStringRequest } from 'skyflow-common'; -import { ReidentifyStringRequestFields } from 'skyflow-common'; -import { ReidentifyStringResponse } from 'skyflow-common'; -import { RequestMethod } from 'skyflow-common'; -import { ReturnEntitiesType } from 'skyflow-common'; -import { ShiftDates } from 'skyflow-common'; -import { SignedDataTokensOptions } from 'skyflow-common'; -import { SkyflowConfig } from 'skyflow-common'; -import { SkyflowError } from 'skyflow-common'; -import { SkyflowRecordError } from 'skyflow-common'; -import { StringCredentials } from 'skyflow-common'; -import { StringKeyValueMapType } from 'skyflow-common'; -import { TokenCredentials } from 'skyflow-common'; -import { TokenizeRequestType } from 'skyflow-common'; -import { TokenMode } from 'skyflow-common'; -import { Transformation } from 'skyflow-common'; -import { VaultConfig } from 'skyflow-common'; - -export { ApiKeyCredentials } +// @public (undocumented) +export interface ApiKeyCredentials { + // (undocumented) + apiKey: string; +} + +// @public +interface Audio_2 { + // (undocumented) + bleep?: Bleep; + // (undocumented) + outputProcessedAudio?: boolean; + // (undocumented) + outputTranscription?: OutputTranscriptionType; +} export { Audio_2 as Audio } -export { BearerTokenOptions } +// @public (undocumented) +export type BearerTokenOptions = { + ctx?: string | Record; + roleIDs?: string[]; + roleIds?: string[]; + logLevel?: LogLevel; + tokenUri?: string; +}; -export { Bleep } +// @public +export interface Bleep { + // (undocumented) + frequency?: number; + // (undocumented) + gain?: number; + // (undocumented) + startPadding?: number; + // (undocumented) + stopPadding?: number; +} -export { CheckGuardrailsRequest } +// @public +export class CheckGuardrailsRequest { + constructor(text: string, fields?: CheckGuardrailsRequestFields); + // (undocumented) + get checkToxicity(): boolean | undefined; + set checkToxicity(value: boolean | undefined); + // (undocumented) + get denyTopics(): string[] | undefined; + set denyTopics(value: string[] | undefined); + get text(): string; + set text(value: string); +} + +// @public +export interface CheckGuardrailsRequestFields { + checkToxicity?: boolean; + denyTopics?: string[]; +} -export { CheckGuardrailsRequestFields } +// @public +export class CheckGuardrailsResponse { + constructor(input: { + text: string | null; + validation: GuardrailsValidation; + toxic: boolean | null; + deniedTopic: boolean | null; + }); + readonly deniedTopic: boolean | null; + readonly text: string | null; + readonly toxic: boolean | null; + readonly validation: GuardrailsValidation; +} -export { CheckGuardrailsResponse } +// @public (undocumented) +export interface ConnectionConfig { + // (undocumented) + connectionId: string; + // (undocumented) + connectionUrl: string; + // (undocumented) + credentials?: Credentials; +} -export { ConnectionConfig } +// @public (undocumented) +export type Credentials = TokenCredentials | PathCredentials | StringCredentials | ApiKeyCredentials; -export { Credentials } +// @public +export interface CustomType { + deidentificationType?: DeidentificationType; + destination?: string; + includePrefix?: IncludePrefix; + // (undocumented) + label?: string; + // (undocumented) + match?: string[]; + prefixOverride?: string; +} -export { CustomType } +// @public +export enum DeidentificationType { + // (undocumented) + ENTITY_ONLY = "ENTITY_ONLY", + // (undocumented) + ENTITY_UNIQUE_COUNTER = "ENTITY_UNIQUE_COUNTER", + // (undocumented) + MASK = "MASK", + // (undocumented) + UNKNOWN = "UNKNOWN", + // (undocumented) + VAULT_TOKEN = "VAULT_TOKEN" +} -export { DeidentificationType } +// @public +export enum DeidentifyFileDataFormat { + // (undocumented) + BMP = "BMP", + // (undocumented) + CSV = "CSV", + // (undocumented) + DCM = "DCM", + // (undocumented) + DOC = "DOC", + // (undocumented) + DOCX = "DOCX", + // (undocumented) + GIF = "GIF", + // (undocumented) + JPEG = "JPEG", + // (undocumented) + JPG = "JPG", + // (undocumented) + JSON = "JSON", + // (undocumented) + JSONL = "JSONL", + // (undocumented) + MP3 = "MP3", + // (undocumented) + PDF = "PDF", + // (undocumented) + PNG = "PNG", + // (undocumented) + PPT = "PPT", + // (undocumented) + PPTX = "PPTX", + // (undocumented) + TIF = "TIF", + // (undocumented) + TIFF = "TIFF", + // (undocumented) + TXT = "TXT", + // (undocumented) + WAV = "WAV", + // (undocumented) + XLS = "XLS", + // (undocumented) + XLSX = "XLSX", + // (undocumented) + XML = "XML", + // (undocumented) + ZIP = "ZIP" +} -export { DeidentifyFileRequest } +// @public +export class DeidentifyFileRequest { + constructor(fields?: DeidentifyFileRequestFields); + // (undocumented) + get configuration(): DetectConfiguration | undefined; + set configuration(value: DetectConfiguration | undefined); + // (undocumented) + get configurationId(): string | undefined; + set configurationId(value: string | undefined); + // (undocumented) + get dataFormat(): DeidentifyFileDataFormat | undefined; + set dataFormat(value: DeidentifyFileDataFormat | undefined); + // Warning: (ae-forgotten-export) The symbol "DeidentifyFileSource" needs to be exported by the entry point index.d.ts + // + // (undocumented) + get dataSource(): DeidentifyFileSource | undefined; + set dataSource(value: DeidentifyFileSource | undefined); + // (undocumented) + get file(): File | undefined; + set file(value: File | undefined); + // (undocumented) + get filePath(): string | undefined; + set filePath(value: string | undefined); + // (undocumented) + get outputDirectory(): string | undefined; + set outputDirectory(value: string | undefined); + // (undocumented) + get pollOptions(): PollOptions | undefined; + set pollOptions(value: PollOptions | undefined); + // (undocumented) + get value(): string | undefined; + set value(value: string | undefined); +} -export { DeidentifyFileRequestFields } +// @public +export interface DeidentifyFileRequestFields { + configuration?: DetectConfiguration; + configurationId?: string; + dataFormat?: DeidentifyFileDataFormat; + dataSource?: DeidentifyFileSource; + file?: File; + filePath?: string; + outputDirectory?: string; + pollOptions?: PollOptions; + value?: string; +} -export { DeidentifyFileResponse } +// @public +export class DeidentifyFileResponse implements DetectRunFields { + constructor(input: DetectRunFields); + // (undocumented) + readonly message: string | null; + // (undocumented) + readonly metrics: Metrics | null; + // (undocumented) + readonly output: FileOutput[]; + // (undocumented) + readonly outputType: DeidentifyFileSource | null; + // (undocumented) + readonly runId: string | null; + // (undocumented) + readonly status: DetectRunStatus | null; +} // @public export enum DeidentifySourceType { @@ -148,12 +241,39 @@ export enum DeidentifySourceType { SKYFLOW_ID = "SKYFLOW_ID" } -export { DeidentifyStringRequest } +// @public +export class DeidentifyStringRequest { + constructor(text: string, fields?: DeidentifyStringRequestFields); + // (undocumented) + get configuration(): DetectConfiguration | undefined; + set configuration(value: DetectConfiguration | undefined); + // (undocumented) + get configurationId(): string | undefined; + set configurationId(value: string | undefined); + get text(): string; + set text(value: string); +} -export { DeidentifyStringRequestFields } +// @public +export interface DeidentifyStringRequestFields { + configuration?: DetectConfiguration; + configurationId?: string; +} -export { DeidentifyStringResponse } +// @public +export class DeidentifyStringResponse { + constructor(input: { + processedText: string | null; + entities: DetectedEntity[]; + metrics: Metrics | null; + }); + readonly entities: DetectedEntity[]; + readonly metrics: Metrics | null; + readonly processedText: string | null; +} +// Warning: (ae-forgotten-export) The symbol "BaseDeleteRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class DeleteRequest extends BaseDeleteRequest { constructor(table: string, deleteIds: Array); @@ -165,6 +285,8 @@ export class DeleteRequest extends BaseDeleteRequest { set table(value: string); } +// Warning: (ae-forgotten-export) The symbol "BaseDeleteResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class DeleteResponse extends BaseDeleteResponse { // @deprecated @@ -178,15 +300,65 @@ export class DeleteResponse extends BaseDeleteResponse { errors: Array | null; } -export { Detect } +// @public +export interface Detect { + // (undocumented) + customTypes?: CustomType[]; + // (undocumented) + entities?: Entity[]; + objectEntities?: ObjectEntity[]; + restrict?: string[]; + returnEntities?: ReturnEntitiesType; + skip?: string[]; +} -export { DetectConfiguration } +// @public +export interface DetectConfiguration { + // (undocumented) + description?: string; + // (undocumented) + detect?: Detect; + // (undocumented) + fileMapping?: FileMapping[]; + // (undocumented) + media?: Media; + // (undocumented) + name?: string; +} -export { DetectedEntity } +// @public +export interface DetectedEntity { + entityScores: Record; + entityType: string | null; + // (undocumented) + location: EntityLocation | null; + token: string | null; + value: string | null; +} -export { DetectRunFields } +// @public +export interface DetectRunFields { + message: string | null; + metrics: Metrics | null; + output: FileOutput[]; + outputType: DeidentifyFileSource | null; + runId: string | null; + status: DetectRunStatus | null; +} -export { DetectRunStatus } +// @public +export enum DetectRunStatus { + // (undocumented) + FAILED = "FAILED", + // (undocumented) + IN_PROGRESS = "IN_PROGRESS", + // (undocumented) + QUEUED = "QUEUED", + // (undocumented) + SUCCESS = "SUCCESS", + // (undocumented) + UNKNOWN = "UNKNOWN" +} // @public (undocumented) export interface DetokenizeData { @@ -196,6 +368,8 @@ export interface DetokenizeData { token: string; } +// Warning: (ae-forgotten-export) The symbol "BaseOptions" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class DetokenizeOptions extends BaseOptions { constructor(); @@ -213,6 +387,8 @@ export class DetokenizeOptions extends BaseOptions { setDownloadUrl(downloadUrl: boolean): void; } +// Warning: (ae-forgotten-export) The symbol "BaseDetokenizeRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class DetokenizeRequest extends BaseDetokenizeRequest { constructor(data: DetokenizeData[]); @@ -221,6 +397,8 @@ export class DetokenizeRequest extends BaseDetokenizeRequest { set data(value: DetokenizeData[]); } +// Warning: (ae-forgotten-export) The symbol "BaseDetokenizeResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class DetokenizeResponse extends BaseDetokenizeResponse { constructor(input: { @@ -235,21 +413,215 @@ export class DetokenizeResponse extends BaseDetokenizeResponse { errors: Array | null; } +// @public +interface Document_2 { + // (undocumented) + pdf?: Pdf; +} export { Document_2 as Document } -export { Entity } - -export { EntityLocation } +// @public +export interface Entity { + deidentificationType?: DeidentificationType; + destination?: string; + entityType?: EntityType; + includePrefix?: IncludePrefix; + prefixOverride?: string; + // (undocumented) + transformation?: Transformation; +} -export { EntityType } +// @public +export interface EntityLocation { + // (undocumented) + endIndex: number | null; + // (undocumented) + endIndexProcessed: number | null; + // (undocumented) + startIndex: number | null; + // (undocumented) + startIndexProcessed: number | null; +} -export { Env } +// @public +export enum EntityType { + // (undocumented) + ACCOUNT_NUMBER = "ACCOUNT_NUMBER", + // (undocumented) + AGE = "AGE", + // (undocumented) + ALL = "ALL", + // (undocumented) + BANK_ACCOUNT = "BANK_ACCOUNT", + // (undocumented) + BLOOD_TYPE = "BLOOD_TYPE", + // (undocumented) + CONDITION = "CONDITION", + // (undocumented) + CORPORATE_ACTION = "CORPORATE_ACTION", + // (undocumented) + CREDIT_CARD = "CREDIT_CARD", + // (undocumented) + CREDIT_CARD_EXPIRATION = "CREDIT_CARD_EXPIRATION", + // (undocumented) + CVV = "CVV", + // (undocumented) + DATE = "DATE", + // (undocumented) + DATE_INTERVAL = "DATE_INTERVAL", + // (undocumented) + DAY = "DAY", + // (undocumented) + DOB = "DOB", + // (undocumented) + DOSE = "DOSE", + // (undocumented) + DRIVER_LICENSE = "DRIVER_LICENSE", + // (undocumented) + DRUG = "DRUG", + // (undocumented) + DURATION = "DURATION", + // (undocumented) + EFFECT = "EFFECT", + // (undocumented) + EMAIL_ADDRESS = "EMAIL_ADDRESS", + // (undocumented) + EVENT = "EVENT", + // (undocumented) + FILENAME = "FILENAME", + // (undocumented) + FINANCIAL_METRIC = "FINANCIAL_METRIC", + // (undocumented) + GENDER = "GENDER", + // (undocumented) + HEALTHCARE_NUMBER = "HEALTHCARE_NUMBER", + // (undocumented) + INJURY = "INJURY", + // (undocumented) + IP_ADDRESS = "IP_ADDRESS", + // (undocumented) + LANGUAGE = "LANGUAGE", + // (undocumented) + LOCATION = "LOCATION", + // (undocumented) + LOCATION_ADDRESS = "LOCATION_ADDRESS", + // (undocumented) + LOCATION_ADDRESS_STREET = "LOCATION_ADDRESS_STREET", + // (undocumented) + LOCATION_CITY = "LOCATION_CITY", + // (undocumented) + LOCATION_COORDINATE = "LOCATION_COORDINATE", + // (undocumented) + LOCATION_COUNTRY = "LOCATION_COUNTRY", + // (undocumented) + LOCATION_STATE = "LOCATION_STATE", + // (undocumented) + LOCATION_ZIP = "LOCATION_ZIP", + // (undocumented) + MARITAL_STATUS = "MARITAL_STATUS", + // (undocumented) + MEDICAL_CODE = "MEDICAL_CODE", + // (undocumented) + MEDICAL_PROCESS = "MEDICAL_PROCESS", + // (undocumented) + MONEY = "MONEY", + // (undocumented) + MONTH = "MONTH", + // (undocumented) + NAME = "NAME", + // (undocumented) + NAME_FAMILY = "NAME_FAMILY", + // (undocumented) + NAME_GIVEN = "NAME_GIVEN", + // (undocumented) + NAME_MEDICAL_PROFESSIONAL = "NAME_MEDICAL_PROFESSIONAL", + // (undocumented) + NONE = "NONE", + // (undocumented) + NUMERICAL_PII = "NUMERICAL_PII", + // (undocumented) + OCCUPATION = "OCCUPATION", + // (undocumented) + ORGANIZATION = "ORGANIZATION", + // (undocumented) + ORGANIZATION_ID = "ORGANIZATION_ID", + // (undocumented) + ORGANIZATION_MEDICAL_FACILITY = "ORGANIZATION_MEDICAL_FACILITY", + // (undocumented) + ORIGIN = "ORIGIN", + // (undocumented) + PASSPORT_NUMBER = "PASSPORT_NUMBER", + // (undocumented) + PASSWORD = "PASSWORD", + // (undocumented) + PHONE_NUMBER = "PHONE_NUMBER", + // (undocumented) + PHYSICAL_ATTRIBUTE = "PHYSICAL_ATTRIBUTE", + // (undocumented) + POLITICAL_AFFILIATION = "POLITICAL_AFFILIATION", + // (undocumented) + PRODUCT = "PRODUCT", + // (undocumented) + PROJECT = "PROJECT", + // (undocumented) + RELIGION = "RELIGION", + // (undocumented) + ROUTING_NUMBER = "ROUTING_NUMBER", + // (undocumented) + SEXUALITY = "SEXUALITY", + // (undocumented) + SSN = "SSN", + // (undocumented) + STATISTICS = "STATISTICS", + // (undocumented) + TIME = "TIME", + // (undocumented) + TREND = "TREND", + // (undocumented) + URL = "URL", + // (undocumented) + USERNAME = "USERNAME", + // (undocumented) + VEHICLE_ID = "VEHICLE_ID", + // (undocumented) + YEAR = "YEAR", + // (undocumented) + ZODIAC_SIGN = "ZODIAC_SIGN" +} -export { FileDataFormat } +// @public (undocumented) +export enum Env { + // (undocumented) + DEV = "DEV", + // (undocumented) + PROD = "PROD", + // (undocumented) + SANDBOX = "SANDBOX", + // (undocumented) + STAGE = "STAGE" +} -export { FileMapping } +// @public +export interface FileMapping { + // (undocumented) + destination?: string; + // (undocumented) + entitiesDestination?: string; + // (undocumented) + objectEntitiesDestination?: string; + // (undocumented) + outputFileNameSuffix?: string; + // (undocumented) + source?: string; +} -export { FileOutput } +// @public +export interface FileOutput { + processedFile: string | null; + // (undocumented) + processedFileExtension: DeidentifyFileDataFormat | null; + processedFileType: string | null; +} // @public (undocumented) export class FileUploadOptions extends BaseOptions { @@ -276,6 +648,8 @@ export class FileUploadOptions extends BaseOptions { setSkyflowId(skyflowId: string): void; } +// Warning: (ae-forgotten-export) The symbol "BaseFileUploadRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class FileUploadRequest extends BaseFileUploadRequest { constructor(table: string, columnNameOrSkyflowId: string, columnName?: string); @@ -290,6 +664,8 @@ export class FileUploadRequest extends BaseFileUploadRequest { set table(value: string); } +// Warning: (ae-forgotten-export) The symbol "BaseFileUploadResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class FileUploadResponse extends BaseFileUploadResponse { constructor(input: { @@ -302,15 +678,26 @@ export class FileUploadResponse extends BaseFileUploadResponse { skyflowId: string; } -export { generateBearerToken } +// Warning: (ae-forgotten-export) The symbol "TokenResponse" needs to be exported by the entry point index.d.ts +// +// @public (undocumented) +export function generateBearerToken(credentialsFilePath: string, options?: BearerTokenOptions): Promise; -export { generateBearerTokenFromCreds } +// @public (undocumented) +export function generateBearerTokenFromCreds(credentials: any, options?: BearerTokenOptions): Promise; -export { generateSignedDataTokens } +// Warning: (ae-forgotten-export) The symbol "SignedDataTokensResponse" needs to be exported by the entry point index.d.ts +// +// @public (undocumented) +export function generateSignedDataTokens(credentialsFilePath: string, options: SignedDataTokensOptions): Promise; -export { generateSignedDataTokensFromCreds } +// @public (undocumented) +export function generateSignedDataTokensFromCreds(credentials: any, options: SignedDataTokensOptions): Promise; -export { GenerateTokenOptions } +// @public (undocumented) +export type GenerateTokenOptions = { + logLevel?: LogLevel; +}; // @public (undocumented) export class GetColumnRequest { @@ -371,6 +758,8 @@ export class GetOptions extends BaseOptions { setReturnTokens(returnTokens: boolean): void; } +// Warning: (ae-forgotten-export) The symbol "BaseGetRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class GetRequest extends BaseGetRequest { constructor(table: string, _ids: Array); @@ -382,6 +771,8 @@ export class GetRequest extends BaseGetRequest { set table(value: string); } +// Warning: (ae-forgotten-export) The symbol "BaseGetResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class GetResponse extends BaseGetResponse { constructor(input: { @@ -394,17 +785,66 @@ export class GetResponse extends BaseGetResponse { errors: Array | null; } -export { GetResponseData } +// @public (undocumented) +export interface GetResponseData { + // (undocumented) + [key: string]: unknown; +} -export { GetRunRequest } +// @public +export class GetRunRequest { + constructor(runId: string); + get runId(): string; + set runId(value: string); +} -export { GetRunResponse } +// @public +export class GetRunResponse implements DetectRunFields { + constructor(input: DetectRunFields); + // (undocumented) + readonly message: string | null; + // (undocumented) + readonly metrics: Metrics | null; + // (undocumented) + readonly output: FileOutput[]; + // (undocumented) + readonly outputType: DeidentifyFileSource | null; + // (undocumented) + readonly runId: string | null; + // (undocumented) + readonly status: DetectRunStatus | null; +} -export { GuardrailsValidation } +// @public +export enum GuardrailsValidation { + // (undocumented) + FAILED = "FAILED", + // (undocumented) + PASSED = "PASSED", + // (undocumented) + UNKNOWN = "UNKNOWN" +} +// @public +interface Image_2 { + // (undocumented) + maskingMethod?: MaskingMethod; + // (undocumented) + outputOcrText?: boolean; + // (undocumented) + outputProcessedImage?: boolean; +} export { Image_2 as Image } -export { IncludePrefix } +// @public +export enum IncludePrefix { + // (undocumented) + INCLUDE_PREFIX_UNSPECIFIED = "INCLUDE_PREFIX_UNSPECIFIED", + // (undocumented) + NO = "NO", + // (undocumented) + YES = "YES" +} // @public (undocumented) export class InsertOptions extends BaseOptions { @@ -435,6 +875,8 @@ export class InsertOptions extends BaseOptions { setUpsertColumn(upsert: string): void; } +// Warning: (ae-forgotten-export) The symbol "BaseInsertRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class InsertRequest extends BaseInsertRequest { constructor(table: string, data: Record[]); @@ -446,6 +888,8 @@ export class InsertRequest extends BaseInsertRequest { set table(value: string); } +// Warning: (ae-forgotten-export) The symbol "BaseInsertResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class InsertResponse extends BaseInsertResponse { // @deprecated @@ -459,8 +903,18 @@ export class InsertResponse extends BaseInsertResponse { insertedFields: Array; } -export { InsertResponseType } +// @public (undocumented) +export interface InsertResponseType { + // (undocumented) + [key: string]: unknown; + // @deprecated (undocumented) + skyflow_id?: string; + // (undocumented) + skyflowId: string; +} +// Warning: (ae-forgotten-export) The symbol "BaseInvokeConnectionRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class InvokeConnectionRequest extends BaseInvokeConnectionRequest { constructor(method: RequestMethod, body?: StringKeyValueMapType, headers?: StringKeyValueMapType, pathParams?: StringKeyValueMapType, queryParams?: StringKeyValueMapType); @@ -476,6 +930,8 @@ export class InvokeConnectionRequest extends BaseInvokeConnectionRequest { queryParams?: StringKeyValueMapType; } +// Warning: (ae-forgotten-export) The symbol "BaseInvokeConnectionResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class InvokeConnectionResponse extends BaseInvokeConnectionResponse { constructor(input: { @@ -491,36 +947,162 @@ export class InvokeConnectionResponse extends BaseInvokeConnectionResponse { metadata?: Record; } -export { isExpired } +// @public (undocumented) +export function isExpired(token: string): boolean; -export { LogLevel } +// @public (undocumented) +export enum LogLevel { + // (undocumented) + DEBUG = "DEBUG", + // (undocumented) + ERROR = "ERROR", + // (undocumented) + INFO = "INFO", + // (undocumented) + OFF = "OFF", + // (undocumented) + WARN = "WARN" +} -export { MaskingMethod } +// @public +export enum MaskingMethod { + // (undocumented) + BLACKBOX = "BLACKBOX", + // (undocumented) + BLUR = "BLUR", + // (undocumented) + NONE = "NONE" +} -export { Media } +// @public +export interface Media { + // (undocumented) + audio?: Audio_2; + // (undocumented) + document?: Document_2; + // (undocumented) + image?: Image_2; +} -export { Metrics } +// @public +export interface Metrics { + // (undocumented) + characterCount: number | null; + duration: number | null; + pages: number | null; + size: number | null; + slides: number | null; + // (undocumented) + wordCount: number | null; +} -export { ObjectDeidentificationType } +// @public +export enum ObjectDeidentificationType { + // (undocumented) + REDACT = "REDACT", + // (undocumented) + UNKNOWN = "UNKNOWN", + // (undocumented) + UNREDACT = "UNREDACT" +} -export { ObjectEntity } +// @public +export interface ObjectEntity { + // (undocumented) + deidentificationType?: ObjectDeidentificationType; + // (undocumented) + entityType?: ObjectEntityType; +} -export { ObjectEntityType } +// @public +export enum ObjectEntityType { + // (undocumented) + ALL = "ALL", + // (undocumented) + FACE = "FACE", + // (undocumented) + LICENSE_PLATE = "LICENSE_PLATE", + // (undocumented) + LOGO = "LOGO", + // (undocumented) + NONE = "NONE", + // (undocumented) + SIGNATURE = "SIGNATURE" +} -export { OrderByEnum } +// @public (undocumented) +export enum OrderByEnum { + // (undocumented) + ASCENDING = "ASCENDING", + // (undocumented) + DESCENDING = "DESCENDING", + // (undocumented) + NONE = "NONE" +} -export { OutputTranscriptionType } +// @public +export enum OutputTranscriptionType { + // (undocumented) + DIARIZED_TRANSCRIPTION = "DIARIZED_TRANSCRIPTION", + // (undocumented) + MEDICAL_DIARIZED_TRANSCRIPTION = "MEDICAL_DIARIZED_TRANSCRIPTION", + // (undocumented) + MEDICAL_TRANSCRIPTION = "MEDICAL_TRANSCRIPTION", + // (undocumented) + NONE = "NONE", + // (undocumented) + TRANSCRIPTION = "TRANSCRIPTION" +} -export { PathCredentials } +// @public (undocumented) +export interface PathCredentials { + // (undocumented) + context?: string | Record; + // (undocumented) + path: string; + // (undocumented) + roles?: Array; + // (undocumented) + tokenUri?: string; +} -export { Pdf } +// @public +export interface Pdf { + // (undocumented) + density?: number; + // (undocumented) + maxResolution?: number; + // (undocumented) + processingMode?: PdfProcessingMode; +} -export { PdfProcessingMode } +// @public +export enum PdfProcessingMode { + // (undocumented) + NONE = "NONE", + // (undocumented) + OCR = "OCR", + // (undocumented) + TEXT_LAYER = "TEXT_LAYER" +} -export { POLL_OPTIONS } +// @public +export const POLL_OPTIONS: { + readonly DEFAULT_WAIT_TIME_SECONDS: 60; + readonly MAX_WAIT_TIME_SECONDS: 300; + readonly DEFAULT_MAX_ATTEMPTS: 23; + readonly MAX_MAX_ATTEMPTS: 23; + readonly MAX_DELAY_SECONDS: 32; +}; -export { PollOptions } +// @public +export interface PollOptions { + maxAttempts?: number; + waitTime?: number; +} +// Warning: (ae-forgotten-export) The symbol "BaseQueryRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class QueryRequest extends BaseQueryRequest { constructor(query: string); @@ -529,6 +1111,8 @@ export class QueryRequest extends BaseQueryRequest { set query(value: string); } +// Warning: (ae-forgotten-export) The symbol "BaseQueryResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class QueryResponse extends BaseQueryResponse { constructor(input: { @@ -541,8 +1125,14 @@ export class QueryResponse extends BaseQueryResponse { fields: Array; } -export { QueryResponseType } +// @public (undocumented) +export interface QueryResponseType { + // (undocumented) + [key: string]: unknown; +} +// Warning: (ae-forgotten-export) The symbol "RedactionRule" needs to be exported by the entry point index.d.ts +// // @public export interface RedactionLevel extends RedactionRule { entityName?: EntityType; @@ -553,13 +1143,80 @@ export interface RedactionLevel extends RedactionRule { tokenGroupName?: never; } -export { RedactionType } +// @public (undocumented) +export enum RedactionType { + // (undocumented) + DEFAULT = "DEFAULT", + // (undocumented) + MASKED = "MASKED", + // (undocumented) + PLAIN_TEXT = "PLAIN_TEXT", + // (undocumented) + REDACTED = "REDACTED" +} -export { ReidentifyFileRequest } +// @public +export enum ReidentifyFileDataFormat { + // (undocumented) + CSV = "CSV", + // (undocumented) + DOCX = "DOCX", + // (undocumented) + JSON = "JSON", + // (undocumented) + JSONL = "JSONL", + // (undocumented) + PPTX = "PPTX", + // (undocumented) + TXT = "TXT", + // (undocumented) + XLSX = "XLSX", + // (undocumented) + XML = "XML" +} + +// @public +export interface ReidentifyFileOutput { + processedFile: string | null; + processedFileExtension: ReidentifyFileDataFormat | null; + processedFileType: string | null; +} + +// @public +export class ReidentifyFileRequest { + // Warning: (ae-forgotten-export) The symbol "ReidentifyFileSource" needs to be exported by the entry point index.d.ts + constructor(dataSource: ReidentifyFileSource, value: string, fields?: ReidentifyFileRequestFields); + // (undocumented) + get dataFormat(): ReidentifyFileDataFormat | undefined; + set dataFormat(value: ReidentifyFileDataFormat | undefined); + get dataSource(): ReidentifyFileSource; + set dataSource(value: ReidentifyFileSource); + // (undocumented) + get redactionLevel(): RedactionRule[] | undefined; + set redactionLevel(value: RedactionRule[] | undefined); + get value(): string; + set value(value: string); +} -export { ReidentifyFileRequestFields } +// @public +export interface ReidentifyFileRequestFields { + dataFormat?: ReidentifyFileDataFormat; + redactionLevel?: RedactionRule[]; +} -export { ReidentifyFileResponse } +// @public +export class ReidentifyFileResponse { + constructor(input: { + status: DetectRunStatus; + outputType: ReidentifyFileSource | null; + output: ReidentifyFileOutput[]; + metrics: Metrics | null; + }); + readonly metrics: Metrics | null; + readonly output: ReidentifyFileOutput[]; + readonly outputType: ReidentifyFileSource | null; + readonly status: DetectRunStatus; +} // @public export enum ReidentifySourceType { @@ -569,20 +1226,71 @@ export enum ReidentifySourceType { PRESIGNED_URL = "PRESIGNED_URL" } -export { ReidentifyStringRequest } - -export { ReidentifyStringRequestFields } +// @public +export class ReidentifyStringRequest { + constructor(text: string, fields?: ReidentifyStringRequestFields); + // (undocumented) + get redactionLevel(): RedactionRule[] | undefined; + set redactionLevel(value: RedactionRule[] | undefined); + get text(): string; + set text(value: string); +} -export { ReidentifyStringResponse } +// @public +export interface ReidentifyStringRequestFields { + redactionLevel?: RedactionRule[]; +} -export { RequestMethod } +// @public +export class ReidentifyStringResponse { + constructor(input: { + processedText: string | null; + metrics: Metrics | null; + }); + readonly metrics: Metrics | null; + readonly processedText: string | null; +} -export { ReturnEntitiesType } +// @public (undocumented) +export enum RequestMethod { + // (undocumented) + GET = "GET", + // (undocumented) + PATCH = "PATCH", + // (undocumented) + POST = "POST", + // (undocumented) + PUT = "PUT" +} -export { ShiftDates } +// @public +export enum ReturnEntitiesType { + // (undocumented) + ALL = "ALL", + // (undocumented) + EXCLUDE_SENSITIVE_DATA = "EXCLUDE_SENSITIVE_DATA", + // (undocumented) + NONE = "NONE" +} -export { SignedDataTokensOptions } +// @public +export interface ShiftDates { + // (undocumented) + maxDays?: number; + // (undocumented) + minDays?: number; +} +// @public (undocumented) +export type SignedDataTokensOptions = { + dataTokens: string[]; + timeToLive?: number; + ctx?: string | Record; + logLevel?: LogLevel; + tokenUri?: string; +}; + +// Warning: (ae-forgotten-export) The symbol "BaseSkyflow" needs to be exported by the entry point index.d.ts // Warning: (ae-forgotten-export) The symbol "VaultEntry" needs to be exported by the entry point index.d.ts // // @public @@ -616,18 +1324,69 @@ export class Skyflow extends BaseSkyflow { vault(vaultId?: string): VaultController; } -export { SkyflowConfig } +// @public (undocumented) +export interface SkyflowConfig { + // (undocumented) + connectionConfigs?: ConnectionConfig[]; + // (undocumented) + logLevel?: LogLevel; + // (undocumented) + skyflowCredentials?: Credentials; + // (undocumented) + vaultConfigs?: VaultConfig[]; +} -export { SkyflowError } +// @public (undocumented) +export class SkyflowError extends Error { + constructor(errorCode: ISkyflowError, args?: Array); + // Warning: (ae-forgotten-export) The symbol "ISkyflowError" needs to be exported by the entry point index.d.ts + // + // (undocumented) + error?: ISkyflowError; +} -export { SkyflowRecordError } +// @public (undocumented) +export interface SkyflowRecordError { + // (undocumented) + error: string; + // (undocumented) + httpCode?: string | number | null; + // @deprecated (undocumented) + request_ID?: string | null; + // (undocumented) + requestId: string | null; + // (undocumented) + requestIndex?: number | null; + // (undocumented) + token?: string | null; +} -export { StringCredentials } +// @public (undocumented) +export interface StringCredentials { + // (undocumented) + context?: string | Record; + // (undocumented) + credentialsString: string; + // (undocumented) + roles?: Array; + // (undocumented) + tokenUri?: string; +} -export { StringKeyValueMapType } +// @public (undocumented) +export interface StringKeyValueMapType { + // (undocumented) + [key: string]: object | string; +} -export { TokenCredentials } +// @public (undocumented) +export interface TokenCredentials { + // (undocumented) + token: string; +} +// Warning: (ae-forgotten-export) The symbol "BaseTokenizeRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class TokenizeRequest extends BaseTokenizeRequest { constructor(values: Array); @@ -636,8 +1395,16 @@ export class TokenizeRequest extends BaseTokenizeRequest { set values(value: Array); } -export { TokenizeRequestType } +// @public (undocumented) +export interface TokenizeRequestType { + // (undocumented) + columnGroup: string; + // (undocumented) + value: string; +} +// Warning: (ae-forgotten-export) The symbol "BaseTokenizeResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class TokenizeResponse extends BaseTokenizeResponse { constructor(input: { @@ -650,9 +1417,21 @@ export class TokenizeResponse extends BaseTokenizeResponse { tokens: Array; } -export { TokenMode } +// @public (undocumented) +export enum TokenMode { + // (undocumented) + DISABLE = "DISABLE", + // (undocumented) + ENABLE = "ENABLE", + // (undocumented) + ENABLE_STRICT = "ENABLE_STRICT" +} -export { Transformation } +// @public +export interface Transformation { + // (undocumented) + shiftDates?: ShiftDates; +} // @public (undocumented) export class UpdateOptions extends BaseOptions { @@ -671,6 +1450,8 @@ export class UpdateOptions extends BaseOptions { setTokens(tokens: Record): void; } +// Warning: (ae-forgotten-export) The symbol "BaseUpdateRequest" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class UpdateRequest extends BaseUpdateRequest { constructor(table: string, data: Record); @@ -682,6 +1463,8 @@ export class UpdateRequest extends BaseUpdateRequest { set table(value: string); } +// Warning: (ae-forgotten-export) The symbol "BaseUpdateResponse" needs to be exported by the entry point index.d.ts +// // @public (undocumented) export class UpdateResponse extends BaseUpdateResponse { constructor(input: { @@ -694,7 +1477,20 @@ export class UpdateResponse extends BaseUpdateResponse { updatedField: InsertResponseType; } -export { VaultConfig } +// @public (undocumented) +export interface VaultConfig { + // (undocumented) + clusterId: string; + // (undocumented) + credentials?: Credentials; + // (undocumented) + env?: Env; + maxRetries?: number; + timeout?: number; + // (undocumented) + vaultId: string; + vaultUrl?: string; +} // (No @packageDocumentation comment for this package) diff --git a/skyvault/samples/detect-api/deidentify-file.ts b/skyvault/samples/detect-api/deidentify-file.ts index 2e36bc77..258ea6a7 100644 --- a/skyvault/samples/detect-api/deidentify-file.ts +++ b/skyvault/samples/detect-api/deidentify-file.ts @@ -9,7 +9,7 @@ import { DeidentifyFileRequest, DeidentifyFileResponse, DeidentifySourceType, - FileDataFormat, + DeidentifyFileDataFormat, } from 'skyflow-node'; /** @@ -63,7 +63,7 @@ async function performDeidentifyFile() { const remoteFileRequest = new DeidentifyFileRequest({ dataSource: DeidentifySourceType.SKYFLOW_ID, // or DeidentifySourceType.PRESIGNED_URL value: '', // or the presigned URL - dataFormat: FileDataFormat.PDF, + dataFormat: DeidentifyFileDataFormat.PDF, configurationId: '', }); const remoteSubmitted = await skyflowClient diff --git a/skyvault/samples/detect-api/get-run.ts b/skyvault/samples/detect-api/get-run.ts index 8c443d44..24f36828 100644 --- a/skyvault/samples/detect-api/get-run.ts +++ b/skyvault/samples/detect-api/get-run.ts @@ -55,7 +55,7 @@ async function performGetRun() { for (const output of run.output) { // output.processedFile is base64 or a presigned URL, per run.outputType // output.processedFileType is e.g. REDACTED_FILE or ENTITIES - // output.processedFileExtension is a FileDataFormat such as PDF + // output.processedFileExtension is a DeidentifyFileDataFormat such as PDF console.log(`${output.processedFileType} (${output.processedFileExtension})`); } console.log('Output type:', run.outputType); diff --git a/skyvault/samples/detect-api/reidentify-file.ts b/skyvault/samples/detect-api/reidentify-file.ts index cc34c485..981e71e0 100644 --- a/skyvault/samples/detect-api/reidentify-file.ts +++ b/skyvault/samples/detect-api/reidentify-file.ts @@ -10,7 +10,7 @@ import { ReidentifyFileRequest, ReidentifyFileResponse, ReidentifySourceType, - FileDataFormat, + ReidentifyFileDataFormat, DetectRunStatus, EntityType, RedactionType, @@ -52,7 +52,7 @@ async function performReidentifyFile() { // formats that support VAULT_TOKEN: TXT, CSV, JSON, JSONL, XML, DOCX, XLSX and PPTX. const content = fs.readFileSync('').toString('base64'); const request = new ReidentifyFileRequest(ReidentifySourceType.BASE64, content, { - dataFormat: FileDataFormat.TXT, + dataFormat: ReidentifyFileDataFormat.TXT, redactionLevel: [ { entityName: EntityType.SSN, redactionType: RedactionType.MASKED }, ], diff --git a/skyvault/src/index.ts b/skyvault/src/index.ts index c9590c28..3d9c5582 100644 --- a/skyvault/src/index.ts +++ b/skyvault/src/index.ts @@ -49,7 +49,8 @@ import { EntityType, DeidentificationType, IncludePrefix, - FileDataFormat, + DeidentifyFileDataFormat, + ReidentifyFileDataFormat, DetectRunStatus, ReturnEntitiesType, MaskingMethod, @@ -86,6 +87,7 @@ import type { EntityLocation, Metrics, FileOutput, + ReidentifyFileOutput, } from "skyflow-common"; import { SkyflowRecordError } from "skyflow-common"; // Detect V2 types whose values depend on the vault type: the skyvault versions. @@ -182,12 +184,14 @@ export { EntityLocation, Metrics, FileOutput, + ReidentifyFileOutput, EntityType, DeidentificationType, IncludePrefix, DeidentifySourceType, ReidentifySourceType, - FileDataFormat, + DeidentifyFileDataFormat, + ReidentifyFileDataFormat, DetectRunStatus, ReturnEntitiesType, MaskingMethod, From 238c790c238279b6d01c3999820cdbae8f7d4eed Mon Sep 17 00:00:00 2001 From: skyflow-bharti Date: Tue, 10 Nov 2026 17:37:10 +0530 Subject: [PATCH 11/11] SK-3076 fix the contract testing --- .github/scripts/api-surface-diff.sh | 125 ++++++++++++++++++++++++++ .github/workflows/contract-tests.yml | 62 +++++++++++-- common/src/detect/controller/index.ts | 2 +- flowvault/api-extractor.json | 2 +- package.json | 2 +- skyvault/api-extractor.json | 2 +- 6 files changed, 184 insertions(+), 11 deletions(-) create mode 100755 .github/scripts/api-surface-diff.sh diff --git a/.github/scripts/api-surface-diff.sh b/.github/scripts/api-surface-diff.sh new file mode 100755 index 00000000..7a058184 --- /dev/null +++ b/.github/scripts/api-surface-diff.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# Summarise how each SDK's public API surface changed relative to the PR's base +# branch, as Markdown for a PR comment and the job summary. +# +# For each published package it diffs the base branch's committed api-report +# against the surface regenerated from this PR's code (temp/, written by +# `api-extractor run`). Using the regenerated file means the comment is right +# even when the PR forgot to re-baseline the committed snapshot; that case is +# called out separately. +# +# Usage: api-surface-diff.sh +# base-ref e.g. origin/main (must already be fetched) +# output.md where to write the Markdown +set -uo pipefail + +BASE_REF="${1:?usage: api-surface-diff.sh }" +OUT="${2:?usage: api-surface-diff.sh }" + +# Keep the comment under GitHub's 65,536-character limit. +MAX_DIFF_LINES_PER_PACKAGE=350 + +# name | committed report | regenerated report | legacy path of the report on +# pre-split base branches (main still has the skyvault report at the repo root) +PACKAGES=( + "skyflow-node|skyvault/api-report/skyflow-node.api.md|skyvault/temp/skyflow-node.api.md|api-report/skyflow-node.api.md" + "skyflow-flowvault-node|flowvault/api-report/skyflow-flowvault-node.api.md|flowvault/temp/skyflow-flowvault-node.api.md|" +) + +{ + echo "" + echo "## Public API surface changes" + echo "" + echo "Compared with \`${BASE_REF#origin/}\`. In the diffs, \`-\` lines were removed" + echo "from the public API and \`+\` lines were added. Removals and signature" + echo "changes are breaking for consumers." + echo "" +} > "$OUT" + +any_changes=0 +stale_snapshot=0 + +for entry in "${PACKAGES[@]}"; do + IFS='|' read -r name report temp legacy <<< "$entry" + + # Prefer the regenerated surface; fall back to the committed snapshot if the + # build never got that far. + current="$temp" + if [ ! -f "$current" ]; then + current="$report" + fi + if [ ! -f "$current" ]; then + { + echo "### $name" + echo "" + echo "No API report was produced for this package (the build failed before api-extractor ran)." + echo "" + } >> "$OUT" + continue + fi + + if [ -f "$temp" ] && [ -f "$report" ] && ! cmp -s "$temp" "$report"; then + stale_snapshot=1 + fi + + base_file="$(mktemp)" + base_note="" + if git cat-file -e "$BASE_REF:$report" 2>/dev/null; then + git show "$BASE_REF:$report" > "$base_file" + elif [ -n "$legacy" ] && git cat-file -e "$BASE_REF:$legacy" 2>/dev/null; then + git show "$BASE_REF:$legacy" > "$base_file" + base_note=" _(base report read from its old location \`$legacy\`)_" + else + : > "$base_file" + base_note=" _(no report on the base branch, so every line is new)_" + fi + + diff_out="$(diff -u --label "base/$name.api.md" --label "pr/$name.api.md" "$base_file" "$current" || true)" + rm -f "$base_file" + + if [ -z "$diff_out" ]; then + { + echo "### $name: no changes" + echo "" + } >> "$OUT" + continue + fi + + any_changes=1 + added="$(printf '%s\n' "$diff_out" | grep -cE '^\+[^+]' || true)" + removed="$(printf '%s\n' "$diff_out" | grep -cE '^-[^-]' || true)" + total_lines="$(printf '%s\n' "$diff_out" | wc -l | tr -d ' ')" + + { + echo "### $name: ${added} added, ${removed} removed${base_note}" + echo "" + echo "
Show diff" + echo "" + echo '```diff' + printf '%s\n' "$diff_out" | head -n "$MAX_DIFF_LINES_PER_PACKAGE" + if [ "$total_lines" -gt "$MAX_DIFF_LINES_PER_PACKAGE" ]; then + echo "" + echo "... $((total_lines - MAX_DIFF_LINES_PER_PACKAGE)) more lines. The full reports are in the \`api-surface-reports\` workflow artifact." + fi + echo '```' + echo "" + echo "
" + echo "" + } >> "$OUT" +done + +if [ "$stale_snapshot" -eq 1 ]; then + { + echo "> **Snapshot out of date.** The committed \`api-report\` does not match the" + echo "> API this PR builds, so the Contract Tests check fails. If the change is" + echo "> intended, run \`npm run contract-snapshot\` and commit the updated reports." + echo "" + } >> "$OUT" +fi + +if [ "$any_changes" -eq 0 ]; then + { + echo "No public API changes in this PR." + echo "" + } >> "$OUT" +fi diff --git a/.github/workflows/contract-tests.yml b/.github/workflows/contract-tests.yml index db487535..7abd3125 100644 --- a/.github/workflows/contract-tests.yml +++ b/.github/workflows/contract-tests.yml @@ -6,6 +6,10 @@ on: - main - release/* +permissions: + contents: read + pull-requests: write + jobs: contract-tests: name: Contract Tests @@ -15,6 +19,11 @@ jobs: - name: Checkout uses: actions/checkout@v4 + # The base branch's committed api-report is the "before" side of the + # public-API diff posted on the PR. + - name: Fetch base branch + run: git fetch --no-tags --depth=1 origin "+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}" + - name: Setup Node.js uses: actions/setup-node@v4 with: @@ -26,15 +35,52 @@ jobs: # Builds every package, bundles the two published dist/ surfaces, then # verifies BOTH SDKs' public-API snapshots (skyflow-node + skyflow-flowvault-node). + # The snapshots include the shared skyflow-common surface, so a change in + # common/ fails here too. Fails if the committed api-report is out of date. - name: Verify API surface snapshots (both SDKs) + id: verify run: npm run contract-snapshot-verify - - name: Show API surface diff + # Runs whether or not the snapshot check passed, so every PR gets a + # comment showing exactly what changed in the public API relative to the + # base branch (a snapshot that was re-baselined in the PR still passes the + # check above, but reviewers still need to see the change). + - name: Diff API surface against base branch + if: always() + run: | + bash .github/scripts/api-surface-diff.sh "origin/${{ github.base_ref }}" api-surface-diff.md + cat api-surface-diff.md >> "$GITHUB_STEP_SUMMARY" + + - name: Post API surface diff on the PR + if: always() + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const marker = ''; + const body = fs.readFileSync('api-surface-diff.md', 'utf8'); + const { owner, repo } = context.repo; + const issue_number = context.issue.number; + try { + const comments = await github.paginate(github.rest.issues.listComments, { + owner, repo, issue_number, per_page: 100, + }); + const existing = comments.find(c => c.body && c.body.includes(marker)); + if (existing) { + await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body }); + } else { + await github.rest.issues.createComment({ owner, repo, issue_number, body }); + } + } catch (err) { + // Forked PRs run with a read-only token; the diff is still in the job summary. + core.warning(`Could not post the API surface comment: ${err.message}`); + } + + - name: Show snapshot mismatch if: failure() run: | - echo "### API surface changes detected ###" - echo "Lines prefixed with '-' were REMOVED from the public API." - echo "Lines prefixed with '+' were ADDED to the public API." + echo "### Committed API snapshot does not match the built API ###" + echo "If this change is intended, run 'npm run contract-snapshot' and commit the updated api-report files." echo "" echo "===== skyflow-node (skyvault) =====" diff -u skyvault/api-report/skyflow-node.api.md skyvault/temp/skyflow-node.api.md || true @@ -42,12 +88,14 @@ jobs: echo "===== skyflow-flowvault-node (flowvault) =====" diff -u flowvault/api-report/skyflow-flowvault-node.api.md flowvault/temp/skyflow-flowvault-node.api.md || true - - name: Upload API surface diffs on failure - if: failure() + - name: Upload API surface reports + if: always() uses: actions/upload-artifact@v4 with: - name: api-surface-diffs + name: api-surface-reports path: | + api-surface-diff.md skyvault/temp/skyflow-node.api.md flowvault/temp/skyflow-flowvault-node.api.md retention-days: 7 + if-no-files-found: ignore diff --git a/common/src/detect/controller/index.ts b/common/src/detect/controller/index.ts index cc966dfe..1dda7e5d 100644 --- a/common/src/detect/controller/index.ts +++ b/common/src/detect/controller/index.ts @@ -165,7 +165,7 @@ export abstract class BaseDetectController/dist/index.d.ts", - "bundledPackages": [], + "bundledPackages": ["skyflow-common"], "compiler": { "tsconfigFilePath": "/tsconfig.dts.json" }, diff --git a/package.json b/package.json index be6621ae..32170ddd 100644 --- a/package.json +++ b/package.json @@ -13,7 +13,7 @@ "build:clean": "tsc -b --clean", "build:dist": "npm run build:dist --workspace skyvault && npm run build:dist --workspace flowvault", "contract-snapshot": "npm run build && npm run build:dist && npm run contract-snapshot --workspace skyvault && npm run contract-snapshot --workspace flowvault", - "contract-snapshot-verify": "npm run build && npm run build:dist && npm run contract-snapshot-verify --workspace skyvault && npm run contract-snapshot-verify --workspace flowvault", + "contract-snapshot-verify": "npm run build && npm run build:dist && sh -c 'npm run contract-snapshot-verify --workspace skyvault; a=$?; npm run contract-snapshot-verify --workspace flowvault; b=$?; exit $((a | b))'", "test": "npm run test --workspaces --if-present", "eslint": "eslint '**/*.{js,ts}' --max-warnings 0", "prettier": "prettier --list-different '**/*.{js,ts}'", diff --git a/skyvault/api-extractor.json b/skyvault/api-extractor.json index 0f7db7c9..fe418aa7 100644 --- a/skyvault/api-extractor.json +++ b/skyvault/api-extractor.json @@ -2,7 +2,7 @@ "$schema": "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json", "projectFolder": ".", "mainEntryPointFilePath": "/dist/index.d.ts", - "bundledPackages": [], + "bundledPackages": ["skyflow-common"], "compiler": { "tsconfigFilePath": "/tsconfig.dts.json" },